{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1.导论"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "什么是机器学习？机器学习的一个重要的目标就是利用数学模型来理解数据，发现数据中的规律，用作数据的分析和预测。数据通常由一组向量组成，这组向量中的每个向量都是一个样本，我们用$x_i$来表示一个样本，其中$i=1,2,3,...,N$,共N个样本，每个样本$x_i=(x_{i1},x_{i2},...,x_{ip},y_i)$共p+1个维度，前p个维度的每个维度我们称为一个特征，最后一个维度$y_i$我们称为因变量(响应变量)。特征用来描述影响因变量的因素，如：我们要探寻身高是否会影响体重的关系的时候，身高就是一个特征，体重就是一个因变量。通常在一个数据表dataframe里面，一行表示一个样本$x_i$，一列表示一个特征。      \n",
    "根据数据是否有因变量，机器学习的任务可分为：**有监督学习**和**无监督学习**。\n",
    "   - 有监督学习：给定某些特征去估计因变量，即因变量存在的时候，我们称这个机器学习任务为有监督学习。如：我们使用房间面积，房屋所在地区，环境等级等因素去预测某个地区的房价。          \n",
    "   - 无监督学习：给定某些特征但不给定因变量，建模的目的是学习数据本身的结构和关系。如：我们给定某电商用户的基本信息和消费记录，通过观察数据中的哪些类型的用户彼此间的行为和属性类似，形成一个客群。注意，我们本身并不知道哪个用户属于哪个客群，即没有给定因变量。      \n",
    "\n",
    "![jupyter](./1.1.png)        \n",
    "\n",
    "根据因变量的是否连续，有监督学习又分为**回归**和**分类**：\n",
    "   - 回归：因变量是连续型变量，如：房价，体重等。\n",
    "   - 分类：因变量是离散型变量，如：是否患癌症，西瓜是好瓜还是坏瓜等。\n",
    "\n",
    "为了更好地叙述后面的内容，我们对数据的形式作出如下约定：      \n",
    "第i个样本：$x_i=(x_{i1},x_{i2},...,x_{ip},y_i)^T,i=1,2,...,N$     \n",
    "因变量$y=(y_1,y_2,...,y_N)^T$        \n",
    "第k个特征:$x^{(k)}=(x_{1k},x_{2k},...,x_{Nk})^T$     \n",
    "特征矩阵$X=(x_1,x_2,...,x_N)^T$\n",
    "\n",
    "在学习机器学习中，我们经常使用scikit-learn简称sklearn工具库来探索机器学习项目，下面我们开始使用sklearn来演示这几个具体的概念："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-14T11:32:35.687512Z",
     "start_time": "2021-03-14T11:32:34.337510Z"
    }
   },
   "outputs": [],
   "source": [
    "# 引入相关科学计算包\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline \n",
    "plt.style.use(\"ggplot\")      \n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.1 回归"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "首先，我们先来看看有监督学习中回归的例子，我们使用sklearn内置数据集Boston房价数据集。sklearn中所有内置数据集都封装在datasets对象内：\n",
    "返回的对象有：\n",
    "   - data:特征X的矩阵(ndarray)\n",
    "   - target:因变量的向量(ndarray)\n",
    "   - feature_names:特征名称(ndarray)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-14T11:33:11.644406Z",
     "start_time": "2021-03-14T11:33:11.150401Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CRIM</th>\n",
       "      <th>ZN</th>\n",
       "      <th>INDUS</th>\n",
       "      <th>CHAS</th>\n",
       "      <th>NOX</th>\n",
       "      <th>RM</th>\n",
       "      <th>AGE</th>\n",
       "      <th>DIS</th>\n",
       "      <th>RAD</th>\n",
       "      <th>TAX</th>\n",
       "      <th>PTRATIO</th>\n",
       "      <th>B</th>\n",
       "      <th>LSTAT</th>\n",
       "      <th>Price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.00632</td>\n",
       "      <td>18.0</td>\n",
       "      <td>2.31</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.538</td>\n",
       "      <td>6.575</td>\n",
       "      <td>65.2</td>\n",
       "      <td>4.0900</td>\n",
       "      <td>1.0</td>\n",
       "      <td>296.0</td>\n",
       "      <td>15.3</td>\n",
       "      <td>396.90</td>\n",
       "      <td>4.98</td>\n",
       "      <td>24.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.02731</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.07</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.469</td>\n",
       "      <td>6.421</td>\n",
       "      <td>78.9</td>\n",
       "      <td>4.9671</td>\n",
       "      <td>2.0</td>\n",
       "      <td>242.0</td>\n",
       "      <td>17.8</td>\n",
       "      <td>396.90</td>\n",
       "      <td>9.14</td>\n",
       "      <td>21.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.02729</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.07</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.469</td>\n",
       "      <td>7.185</td>\n",
       "      <td>61.1</td>\n",
       "      <td>4.9671</td>\n",
       "      <td>2.0</td>\n",
       "      <td>242.0</td>\n",
       "      <td>17.8</td>\n",
       "      <td>392.83</td>\n",
       "      <td>4.03</td>\n",
       "      <td>34.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.03237</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.18</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.458</td>\n",
       "      <td>6.998</td>\n",
       "      <td>45.8</td>\n",
       "      <td>6.0622</td>\n",
       "      <td>3.0</td>\n",
       "      <td>222.0</td>\n",
       "      <td>18.7</td>\n",
       "      <td>394.63</td>\n",
       "      <td>2.94</td>\n",
       "      <td>33.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.06905</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.18</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.458</td>\n",
       "      <td>7.147</td>\n",
       "      <td>54.2</td>\n",
       "      <td>6.0622</td>\n",
       "      <td>3.0</td>\n",
       "      <td>222.0</td>\n",
       "      <td>18.7</td>\n",
       "      <td>396.90</td>\n",
       "      <td>5.33</td>\n",
       "      <td>36.2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      CRIM    ZN  INDUS  CHAS    NOX     RM   AGE     DIS  RAD    TAX  \\\n",
       "0  0.00632  18.0   2.31   0.0  0.538  6.575  65.2  4.0900  1.0  296.0   \n",
       "1  0.02731   0.0   7.07   0.0  0.469  6.421  78.9  4.9671  2.0  242.0   \n",
       "2  0.02729   0.0   7.07   0.0  0.469  7.185  61.1  4.9671  2.0  242.0   \n",
       "3  0.03237   0.0   2.18   0.0  0.458  6.998  45.8  6.0622  3.0  222.0   \n",
       "4  0.06905   0.0   2.18   0.0  0.458  7.147  54.2  6.0622  3.0  222.0   \n",
       "\n",
       "   PTRATIO       B  LSTAT  Price  \n",
       "0     15.3  396.90   4.98   24.0  \n",
       "1     17.8  396.90   9.14   21.6  \n",
       "2     17.8  392.83   4.03   34.7  \n",
       "3     18.7  394.63   2.94   33.4  \n",
       "4     18.7  396.90   5.33   36.2  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn import datasets\n",
    "boston = datasets.load_boston()     # 返回一个类似于字典的类\n",
    "X = boston.data\n",
    "y = boston.target\n",
    "features = boston.feature_names\n",
    "boston_data = pd.DataFrame(X,columns=features)\n",
    "boston_data[\"Price\"] = y\n",
    "boston_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-14T11:33:15.385401Z",
     "start_time": "2021-03-14T11:33:15.228402Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "e:\\a_projects\\pyprojects\\py_38_project\\venv\\lib\\site-packages\\seaborn\\_decorators.py:36: FutureWarning: Pass the following variables as keyword args: x, y. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAEaCAYAAAD3+OukAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABslklEQVR4nO29eXyU1b34/z7PMzOZmezJJEAISxJAEAEFFcG6h7oAghZorXa1t7XS2tra6u29bW2vtra9LleLt/ptr1t/rXUrCrhUxB13BQQR2SEEyELIOpNZnvP742QmM1knMEkmmfN+vXiFPHmemXNmnud8PuezCimlRKPRaDQpiTHYA9BoNBrN4KGFgEaj0aQwWghoNBpNCqOFgEaj0aQwWghoNBpNCqOFgEaj0aQwWghoUoYHH3wQm8022MPQaJIKLQQ0Q5Kvf/3rCCEQQmCz2Rg3bhzXXHMNtbW13V7zxS9+kQMHDgzgKDtz8803I4Rg2bJlnf5ms9l48MEHY4599NFHLFmyhMLCQhwOB+PGjeP73/8+hw8fjpzzzW9+k0mTJtHc3Bxz7XXXXcfYsWM5evRof0xFM0zQQkAzZDnrrLM4ePAge/bs4e677+bJJ5/kq1/9aqfzpJQEAgFcLhcjRowYhJHG4nQ6eeKJJ3j77bd7PO/5559nzpw52Gw2Vq1axY4dO7jvvvtYv349p556Kvv37wfgf/7nfwiFQvzoRz+Kufbee+/l4YcfJicnpz+noxnqSI1mCPK1r31NXnDBBTHHbrnlFmkYhrz33nulaZpy3bp18uSTT5Z2u10+++yz8oEHHpCmacZc8/7778sLL7xQZmZmyvT0dHnaaafJt99+O/L3f/3rX3Lu3LnS6XTKoqIi+fWvf13W1NQc87h/+ctfyrKyMrlkyRJ55plnxvzNNE35wAMPSCmlbGlpkSNGjJAXX3xxp9eor6+XhYWFcuHChZFjb775pjRNU65atUpWV1fLkSNHyhtuuOGYx6lJHfROQDNscLlcWJZFMBjEsixuvPFG7rjjDj799FNOPfXUTudv2bKFs88+m9zcXNatW8dHH33E9ddfj2VZAKxbt45FixbxpS99iU2bNrFy5Ur27NnD5ZdfjuxDtZXw60Vz22238d577/Hkk092ec2//vUvDh8+zM9+9rNOf8vKyuJ73/sea9asiZh65s6dy0033cTVV1/NlVdeSWFhIbfcckvcY9SkMIMthTSaY6HjTmDLli2ytLRUzp49Wz7wwAMSkK+99lrMNR13AldddZWcPn26DIVCXb7HOeecI2+88caYY3v37pWA/Oijj6SUUvr9fvn73/9eTpkyRRYUFMgLL7xQPvLII7K+vl76/X756KOPyp/85CeR68M7ASml/OEPfygnTJgg/X6/lDJ2J/C73/1OAvLIkSNdju3JJ5+UgHz33XcjxwKBgBw3bpwUQshNmzb19PFpNBH0TkAzZHnllVfIyMjA5XJx0kknUVpayt/+9rfI30877bQer//ggw+44IILMIyuH4P33nuPu+66i4yMjMi/E088EYDt27cD8Pe//53169dz77338txzz3Huuefyhz/8gezsbNLT07nrrrv44he/2OXr//znP6empob//d//PZbpd+L555+noqKCtLQ0XnvttYS8pmb4o+PlNEOW2bNn89BDD2Gz2SgqKsLhcADw2muvYZomTqfzuF4/bFL6yle+0ulvI0eOBGDhwoUxzuhZs2Zx0003UVNTg9PpJCMjo9vXz8vL4z/+4z/49a9/zde+9rWYv02aNAmAzZs3c9ZZZ3W6dsuWLRiGwYQJEwCoqqri6quv5sYbb6SwsJCf/vSnzJs3L/I6Gk13aCGgGbK4XK7IIngszJo1i5deegnLsrrcDZx66qls2bKlx/fIzc3t8rjH44lrDN///vdZsWIFt956a8zxz3/+8xQUFPDb3/62kxBoaGjgj3/8I/Pnz4+8/7e+9S2Ki4u5+eabsdlsrF69mq985Su8+eabOjdC0yPaHKRJWX7605+yfft2rrzySt5//3127tzJ448/zltvvQXAr3/9a55++ml+9KMfsWHDBnbu3Mnzzz/P1VdfjdfrTcgY0tLS+M1vfsPdd98d40B2u9088MADvPTSS1xxxRW8++677N+/nxdeeIHzzz8fh8PBihUrALjvvvtYu3Ytf/3rX7Hb7QgheOCBB9i+fTu/+c1vEjJOzfBFCwFNyjJt2jReeeUVqqurOeecczj55JO5/fbbMU0TgPPOO49169axadMmzjrrLKZPn871119PZmYmdrs9YeP40pe+xIwZMzpFHM2fP5/169fj9/u55JJLKCsr49/+7d8444wzeP/99xkzZgzbt2/nxz/+Mb/73e+YMmVK5Nri4mJWrFjBf/3Xf/H+++8nbKya4YeQHe88jUaj0aQMeieg0Wg0KYwWAhqNRpPCaCGg0Wg0KYwWAhqNRpPCaCGg0Wg0KcyQzCKprKzst9f2eDzU1NT02+snM3ruqTf3VJ03pN7ci4qKujyudwIajUaTwmghoNFoNCmMFgIajUaTwmghoNFoNCmMFgIajUaTwgxYdNDy5ctxOp0YhoFpmtx22200NTVx5513Ul1dTUFBAddff32P9df7GwEY+/eTsWcPlseDt6CArgorCcBVXY1RU4P0eMAwEFVVPV4z2ESPOd65JfN8wiRyvDYgd9MmjH37sMaOpW76dIL9cP2xjHmofS+aocOAhoj+8pe/JCsrK/L7ypUrmTZtGosXL2blypWsXLmSq666aiCHFEEAWevXI158EbO5GdM0sS9cSMPcuTEPW/g8Vq2CUAhHbS1Mm4a/rg7T7+/ymsGm45i7mxtSxndekhD3vOIhGMTzyCNw663g82E6nXj+4z+o+cpX4hIENojr+mMZc0LnqdF0YFDNQe+99x7nnHMOAOeccw7vvffeoI3FVV0decgA9XPVKnW8m/NsPh/s3AmrV2OOHt3tNYNNvHMzKiriOi9ZiHde8WC++WZkAQfUz1tvJXfTpriuz920Ka7rj2XMiZynRtORAd0JhLsnzZs3j/Lycurr6yOdkXJycqivr+/yurVr17J27VoAbrvttri7NvUFY+9ehN2OYRgxbQnTWlpwR71f+DzsdkRTExgGWBZGMIi97bqO1ww20WOOptPcNmzAGcd5yUK884oHsWcPorUVhGg/2NqKsX8/nvPP730s+/fHdf2xjDmR8+yIzWbrl+dpKJDKc49mwITAf/3Xf5GXl0d9fT233HJLp+w1IQQi+gGKory8nPLy8sjv/ZHl53a7sQcCOO12fGFtzjQJuN20RL1f+DxCIWw2G4Zlgc1GyGYj5PN1ec1gEz3mCF2MszA/n9Y4zksW4p1XPIwYPx7S0to1eQCnE2vMmLjut4IxYxBxXH8sY07kPDuSalmz0aTa3Ac9YzgvLw+A7OxsTjvtNHbs2EF2djZ1dXUA1NXVxfgLBhpvQQEsXAhtXaUwTVi4UB3v5ryg0wllZbBgAaEDB7q9ZrCJd25WcXFc5yUL8c4rHkJnngn/8R8Q3gU6nfAf/0Hd9OlxXV83fXpc1x/LmBM5T42mIwPSWczn8yGlxOVy4fP5uOWWW1iyZAkff/wxmZmZEcdwU1NTXI7h/qodJIACr5eWFI0O8ng81NbUDKkolERFzXg8Ho7W1AxIdJABZO3ejVFRgVVcTENJCVYX50XTX9FBqaYNR5Nqc+9uJzAgQuDw4cP893//NwChUIjPfe5zXH755TQ2NnLnnXdSU1PTpxDRwS4gN1zD9VLtoYhmoObeMdInrNUPVqSP/s5TZ+6DKgQSzWAKgWR7iBNJeO7DVcj1xEAtCO7qaux33NHZvv+jH9EyCOadVFsIo0m1uXcnBIZkKenBpNtwvYkTB+UhTjQ6Jr1/MWpqYgUAQCikjg+D+0cz9NBlI/pIjw/xMEDHpPcvlsfT7uANY5rquEYzCGgh0EeG+0M83IXcYKMjfTTJhjYH9RFvQQH2hQs7+QSGy0NseTyYptnJZj1chNxgI4GGuXNxTZyYUj4XTfKihUAfGe4P8XAXcsmABOU/0p+pJgnQQuAYGM4P8XAXchqNJhbtE9DEkIrhoRpNKqN3App2hlgpaY1Gc/zonYAmwlArJa3RaI4fLQQGAYHKHM3YuhV3dTVd104dBKqqdHioRpNiaHPQAJPUGbmFhSoaSIeHajQpg94JDDDhjFxbczP2o0exNTcnjcllqJWS1mg0x4/eCQwwZk0NjspK2LsXpEQIgWPcOKxkqB0jhA4P1WhSDC0EBhjT6YTKSggXb5USKivV8SRgOOdAaDSazmhz0AATDARg/vz2frF2O8yfr44nAUnrtNZoNP2C3gkMMFZuLv6GBsxlyzB8Piynk9ChQ1i5uYM9NJ0noNGkIHonMMB4CwqgvJzQ7t0EKioI7d4N5eVJ4XzVeQIaTeqhdwIDTFLX5ukpTyAJhJRGo0k8Kb0TGCz7d9j52jxlCgDpyWJ/D+cJRKPzBDSaYU3K7gS6S9pi4cJBff/BtL9H8gR0GWmNJmVIWSHQXRtF45RTwOUatPcf1F7FQzBPQFc91WiOj5QVAt21UaSqCsaNG7T3H2z7+1DKE0jG3ZRGM9RIWZ9Ad72CKSwc1PfX9vf46XY3leTRTDoXQ5NMpKwQ6K7ht1VcnJDX7+1B1w3Hj58ed1NJSnj3Yr/jDswHHsB+xx1krV+vBYFm0EhZc1B3oZr54vgfx3jMFEkdKjpEsDwezCFW9TQpfUGalCZldwLQbv9umjKFljgX4Hi28vGaKY7l/TXtDMXd1FDcvWiGNym7EzgW4nVEJqvTd7gxFHdTQ3H3ohnepPROoK/Eq+Frp+/AMdR2U0Nx96IZ3uidQB+IV8P3FhSoxLMhmnSlY+/7j6G4e9EMb7QQ6APxbuWH8oOeyrH3AyX8hlIuhmb4o4VAH+iLhj9UH/SUjV7RZbQ1KYoWAn1gKGv48ZKqTu1uy2gPd+GnSXm0EOgjQ1XDj5eUjV7RZbQ1KYqODtLEkLLRK7qMtiZF0TsBTQypYPLqCl1GW5OqDKgQsCyLm266iby8PG666Saqqqq46667aGxspLS0lO9///vYbFouDTbD3eTVJUOwjLZGkwgG1Bz07LPPMnr06Mjvf/3rX5k/fz733HMP6enprFu3biCHk1LoypW9M9QSzzSaRDBgQqC2tpYPP/yQCy64AAApJVu2bOGMM84A4Nxzz+W9994bqOGkFLpypUaj6Y4Bs708+OCDXHXVVXi9XgAaGxtxu90qEgXIy8vjyJEjXV67du1a1q5dC8Btt92Gpx+ddTabrV9ffzAw9u9HvPgi2O3qH8CLL5J2yilYY8ZEzhuOc4+XVJ17qs4bUnvu0QyIEPjggw/Izs6mtLSULVu29Pn68vJyysvLI7/X9GPFRY/H06+vPxhk7NmD2dzc6Xhozx6aolppDse5x0uqzj1V5w2pN/eioqIujw+IENi2bRvvv/8+H330EX6/H6/Xy4MPPkhLSwuhtuzMI0eOkJeXNxDDSTlSNvZfo9H0yoD4BL785S/zpz/9iRUrVvDDH/6Qk046ieuuu46pU6fy9ttvA/DKK69w6qmnDsRwkoaBctambOy/RqPplUGNx7zyyiu56667ePTRRykpKeH8888fzOEMKANZqC1VY/81Gk3vDLgQmDp1KlOnTgVgxIgR/Pa3vx3oISQFA12oLSVj/zUaTa/oshGDxEC3GdR5AhqNpit0eu4gMZDO2lTuEaDRaHpG7wQGiYF01sbbFhP0jkGjSTX0TiAO+qPj1EA6a8OmJ5vPh/D5kE4nQaezc5lk3VhFo0k5tBDohUSZUroTJAPhrJUeD47aWti5E6RECIGjrIyWDqYn3VhFo0k9tBDohURE8Qy6Td4wYNo02LsXAgGw2dTvRgdroG6sotGkHFoI9EIi2i0Odt9eUVWFv64Oc9kyDJ8Py+kkdOAAoqoK8vPbTww3VtGZxRpNyqCFQC8kIorHqKkBhwNz9OiYRXigNGzL48H0+wnt2EFkFl3MQTdWGRj6w8ek0RwrWgj0gregAPtxLoyysBBHbi489hgEAph2O+aCBbQUFvZ4XTyLRTzneAsKsF92GebGjRheL5bLRWjGjM5z0I1V+p1BNw1qNB3QQqAXEhLFY1nw8ccQDKrfg0H1u2V1e0k8i0WfFpRgELF1KzQ1ITIyoC1ru6v56szi/iNsGrQ1N7dHamnnu2YQ0UIgDo53YRQ1Nfjz87Glp8eEaIoezEHx+BHi9TWEzwvabJCTow7qhWdQMGtqcFRWKid9OFJr3Dgs7XzXDBI6WawjUiY8WcryeMA0CTqdBHJyCObmYk6ciL2lpdv3iKesRLylJwa6RIWme0ynEyorQbbt1aSEykp1XKMZBPROIAoBmC++iP3xxxNqr43xKzgcyj/w9tv4t2/H3s17dOmQzsjA5nSSsXUrlseDbBMuvTmtdT+B5CEYCGCfPx9Wr1bhunY7zJ9PMBAY7KFpUhQtBKLor1DOaL9C2sGDWA89BOnp2I8e7dYm3MkhnZGBY/p0/Pffj9kmoLjssriieRLh3NYkBis3F39DQ2y47qFDWLm5gz00TYqihUAUkVDOUaOOOZSzt2gd0dCA7fBhrIqKHm3CHR3SNqcT//33xwgo26OPYl19NfLb3ybk8xHqxmmt+wkkD96CAuzl5YRWrSKkBbImCdBCIApZWIjIzMR86imE3Y4pBLbFi2keOTKu67uL1mmcO5fMtuPm6afD5s0YDgeWy9WjTTjaIZ2xdStmlDnHcfgw7N2LGD+ewMGDGAsX0jRlSrcLu476SQ60QNYkG9oxHI1lqQVaCMSuXbB9O+L//o+0zz6Ly0HcnTkpa/fuyHFZXQ2XXAI+H0YoFLdNOOxcBrD5fCq6xGbDcjp7rAqqST7CArlpyhRatADQDDJ6JxCFqKkBtxuamiArSzlPTRPj3XdxlZb26hfoNgqnokJt/QELEFu3IubPh9xcQjk5cdmEo+36wudT9X8WLCB04ED7++gwQ41G00e0EIjC8nggGMQyDHA41EG7HSstLa4FttsonOLiSBRP6MABzHPOQa5ZgzVlCqG6urhswjFmhH37EB9+SKiiArze9vfR0T5DAl02QpNMaCEQhbeggMylS2HrVvD7lalmwQKlqcexwHYXhdNQUkJm+LjXi980MW+5hUBWVrfO3K4ImxG8BQVktbbCrl3qD9q5OGTQZSM0yYYWAlFIILRoEZbDgfHuu1hpaYQOHYLy8rgW2J6cfh2PN8Sx8HenMca8Xl0dNrudkM+Hq7paa5VJzmBXlNVoOqKFQEcMg6OzZ5NVWIhRUYGYN4+GkpK4F9boKJwuF/GoB10A7ro6XHv3Io4cIVBSQkNJifIb0LPGKFE7j6zt25GrVmGEQhhaq0x6ElGaXKNJJFoIdMSyyHnnnchOQL78Mpnl5Z0Kt8VT3bOnRVwAWR9+iPP99+Hee6G1lbT8fPJvuIHaxYtxJrB2kCZ50NnbmmRDC4EoBGA+/TTi178Gvz9S8tm/dm1kYY3XptvTAu0tKCB7927S9u6FTz4BpxNaW6G2FuPuu8maMYOgz9erxqi1yqGHzt7WJBtaCEThqq5GPP64cgqDqu2yejXmsmWE2hbWeLXvbhfoujqytm/H9te/IiorYcsWWLAA+dZbUF8PXi9GRQXW5Mm9aox90Sp1REpyoJPFNMmGThZrQwBpBw9CaSnizDMhM1P9IRDAaG2NLKzxVuSMTu6KYJrY7HZYtQppsyEdDlU64rnnEKefjnA4wO3GKi5WmuHChe2v0YXGGM854bllrV+P/Y47MB94APsdd5C1fn1CKqRq+o5OFtMkE3onQPsiaTzxBGzciPD5EOecg7VpE7S2Yp1+emRhjVf77mrbb5s3D3ngAKK5mZDTiZGVhW3qVNi8GdGWPSy/9S0a2xzRvWmM8WqV2neg0Wi6QwsBYpuu2MeNw9q1C+OttxCXX05o6lSOzp4dWVjjtelGL9BmTQ32hgaCL7yAOXIkxtatGEVFWDk5qozE9OnIs8+GmTOxfD6ctbU05+fHVe8nnnO070Cj0XSHFgLELpKyuBgrIwPp8xGaO5ejs2YB4I6ypzfOnYszDptueIF2A8bDD7dnDM+fD2vWYDNN5N69cMklWI88Ao2NYLfjmDWL5vz8hM2vL74DA8javVv5JYqLIyGrGo1meKKFAJ0XyaDTCenpBMaOBXqIBopTi47RxL1e/IC5dCnG2LHIkSORmzYpAQCq/3CCG4zEHZFiWeQ//TTiyScRJSUgBJ7PfY7a8nJCXb+0RqMZ4mghQIfibM3N2L1e5JIlNPYhGqgnOmniXi+h3bux5s1TXcyamtRxIaCsDH+b8DleoiOCghMnwg03IKqqut29mJs3w6OPIkaNgr/9TTWnX72avFtvpebCC7UDU6MZhmghgDLbNM6dS45pIj/6CMswCL3xBplOJyGP57jt6T3WFLr6amxPPIFoakJmZBBcsiQhztqO+QyR9+wpm3jXLjWnVavadyO1tRiPPopr5kztRE4QOlxXk0xoIdCGs7qa0MqV2Nvq8ACwahW2b38b2YcMz+4e8HhrCiVqQTimHUxGhvoZbY4SAoTQTuQEIYCsd9/F3LgRw+vFcrmwz5hBw+mna0GgGRS0EGgjYre329sPhkKEfD6MODM8e8om7o7+6vh1LBFB8oQTEKefDi+8oDKYDQMmTCCYm6vLGiQId20tjvXrI43mTbsds7kZd1lZQoMBNJp40UKgjYjdHsDlwhw9GqO1laDTSUOc0UDdad/uE07Atm3bgJYPPpYaNVZxMYETT8T5058innoKKQTWqFEEv/AFXdYgQTgOHIgIACCSle44/3wtBDSDwoAIAb/fzy9/+UuCwSChUIgzzjiDZcuWUVVVxV133UVjYyOlpaV8//vfx2YbHLkUsdu/8QYOhwMefxyKipCVlWRedhnBkpJeX6M77dtx4ADWACdrHVONGiFomDmT4LhxuE4+GdHYSGDcuEjymiYBBAIQDKrWom3fiyVEwiPCNJp4GZAV126388tf/hKn00kwGOQXv/gFJ598MqtXr2b+/PmceeaZ3H///axbt47Pf/7zAzGkToTt887x45G33IKcMkWFijocONavx/boowRtth61+O60bwIBCIWw+XwInw/pdBJ0OvvVzn48/gbbtm0E2oSHLRgkt7wc3+zZtLQlsKUSiXbi+seOxVFUBBs3qp7WhoExY0bCIsI0mr4yIEJACIHT6QQgFAoRCoUQQrBlyxZ+8IMfAHDuuefy+OOPD5oQALVwykCAQNhBCpijR8NjjyFKSyEnp0ctPkb7djgwi4sxxo0jmJeHw+eDTZtUrSAhcJSV0dLPdvZeext0cU20Sctx+LBqaL9pE2lf/CK2qVNTqldBv3QBMwysuXMRXi+itRWZloacO1f5XzSaQaBPQqCxsZGPPvqIuro6Fi1axJEjR5BSkh+HLdOyLG688UYOHTrEhRdeyIgRI3C73RE7fF5eHkeOHOny2rVr17J27VoAbrvtNjz9uHgara0409PbtflgECElZGZitgkygLSWFtxdjWPhQoyZMxGvvYZYuxa5fz+2tWth2jSorIQjR5Tz+ZRTSM/MxDUQDlcpMV98McY0lLlwIaF581T0Txs2m42MlhaE3Y7w+6GiAmFZ0NKC2dyMfPFF0k45BWvMmP4f8wBjs9k63VfG/v2IF19U31c4YOA4PwNj715EY6Mq/OfzIZxOxOHDpDc14TrhhOOdRp/pat6pQirPPZq4hcAnn3zC7bffTmlpKdu2bWPRokUcOnSIZ555hptuuqnX6w3D4A9/+APNzc3893//N5WVlXEPsry8nPLy8sjvNR0qdiYCgSoNkXHoENbZZ2O99BLBcE+B8ePxmyaEQ0dNk4DbTUs343A3NmJ/+WX1i8+H/ehRxJNPwne+g9y/H8vpJHTgAKHdu2l2Ovs9ZtxdXa2S0qLNVI8/TmDMmJjdjMfjocXtxm6zkTZmDEJKpN8PGzYgq6qQBw7g27OHRpcrwSMcfDweT6f7KmPPHszm5k7nhvbsoekYPwO32429oQHq6toP9nI/9SddzTtVSLW5FxUVdXk8biHw4IMP8sMf/pBp06bxjW98A4AJEyawc+fOPg0kPT2dqVOn8tlnn9HS0kKobZt95MgR8vLy+vRaiSK87Xf85S+IPXvAbsdctgxRUoJ3+nRV6bMPDtaODmLpdCpfwP79BCoq1EHTRBYWDkjT8b6Ei/oKCkifPh1x112IvXvB4YBrr8V66y2orY3ZDQ13+qMLmG4qo0k24hYC1dXVTJs2LfZim41Qx8WlCxoaGjBNk/T0dPx+P5s2bWLRokVMnTqVt99+mzPPPJNXXnmFU089te8zSACu6mpsTzwBO3cq26zXi/zb37C++EXkSSf12cHaVS0ix/TpiDFjsAOWy0VoxgzlGOwhaihRTsnweGzNze2O6fT0LhczZ3U1/nfeIW3iRER2NjQ0IB9+GHHxxcgzziCY4CiWZM6e7Y8FWzeV0SQbcQuB4uJiNmzYwMknnxw59vHHHzM2jqiGuro6VqxYgWVZSCmZM2cOs2bNori4mLvuuotHH32UkpISzj///GOaxPFi1NQgmppARj2KgQCG13tMETydFo+MDJg7F+v11xGNjYiMDJg6FaOuLnZhdjojGrooKIhrlxDPIuorKCB99mxYsQL8foTDgWP5chq7mFd41xByuZDNzYiMDITLhZw8Gf+uXVi5uX36LHqiXxyvCaS/Fuz+ShDUaI6FuIXAV77yFX73u99xyimn4Pf7uf/++/nggw/4yU9+0uu148aN4/e//32n4yNGjOC3v/1t30bcD1geDzIjAxHlJMVux3K5jslk03HxsDmd+O+/X2mSOTnqpFWrcF52GWzdqhZmIXCMG4e/qAjL44mr7EO8JQic1dX4t2/Hce21iPp6ZHY2/u3bcVZXd4pwCu8agk4njtGjkXv3Im02Ql4vlJcn1GwxFJrd6AVbM9yJWwhMmjSJP/zhD7z++us4nU48Hg+/+c1v4ooMSna8BQXYlyzBUV8PbT4BFixoN9msXYtZUoLh8ymnblTj+Wi61MoLCsjYuhWzg9nM1tyM/PBDxPz57RmklZXYvvY1GgoKSN+6tVc7frwlCIy6Ohx2O6xYgQwEVM+CBQsI1dV1WtyidzH+ESOw5edDeTmt/ZAnoJvddE0ym8g0w4+4hUAgECArK4tFixZFjgWDQQKBAPboejtDkLDm7p44kYxDhwhKiX/0aFry80n/7DMcWVnw2GPtC20XC2hPpo2uHIwiGFTVSuvqMJctiwgYf3ExkvickvGWILDZ7bBmTex5a9Zgi/ouO34W4V1MoB8Xof5wvA51kt1Ephl+xJ2hcsstt7Br166YY7t27eLWW29N+KAGAwk0FxQQOu886qZPj7R37HYB7SD4ujVtVFd32RBeLllC6NAh1Vtgxw4CFRWqx0CbzT2uJvJtJQhi6KIpTcjng6Ki9pwAIaCoqL1a6jEQCanduhV3dfUxNa2Pa479RMfxx/iDBpGe7qPBJBHftyY5iXsnsG/fPiZOnBhzbMKECezduzfhg0omQj4f9qIilTkrZbcLaE+mDVlQ0MnB2FhQQKbT2W3kSTxOSf/YsdjKylRUU3hsXTSlCXk8+IuKsGVnx0QHhbrQuOPRRBOlrQ5WpExX4zeXLkXMnDno2nYymsj07mR4E7cQcLvd1NfXkxN2bAL19fWkpaX1x7iShngX0L6aNuJZAHtzSrYUFGCLoylN2M4fXLUKnM4eNe54nLUdz7E1NyP++leyR42ivo/F5gbD8dpttdcxY5AwqLb4ZDSRDQUHvubYiVsIzJ49m//5n//hG9/4BiNGjODw4cM89NBDzJkzpz/HN+jEu4D2FFPekyZ1PAtgvJp0XzTueDTR6HMi9YWkxLZ6NVmzZiW9htjdHF0ffUTg5ZcHVdtNxmSyZNydaBJH3ELgS1/6Eg8//DA/+9nPCAQCOBwOzj33XK644or+HN+gc0wLbV0dtrYOZa7qaoRhdKtJeQsKVJTPgQMQCOAfO5aWPmig8WrS8Z4XT2KZ5fFgZmTgyM/HEAJZVIT89FOstLQhoSF26aj3+bD27h10bTcZk8mScXeiSRxxCwGHw8G3vvUtrr76ahobG8nMzIyNqx/G9GWh9RYUkLV9O3LVKoxQCEcwiPm5zxGcPJnQrl3g9aqTQyHMmhqydu9uD/MMBrGVlWG7+uqEa6Dxhh3Gk1gWLi1h3HknVFUh0tIQ116L/8iRIaEhdqVty/POI/Tmm7EnDtJcki03IRl3J5rE0aMQqKqqorCwEIDDhw/H/M0bXsxQSV8aRdh+KkIh0iorVfP2Dz7AdvbZmPn5+EEJAtPEdDox3ngjNsxz505sTzyRUA20L449Z3U1/k2bMJcubc+L2LQJ55lnRsYTLi3hOPFEjIwMpGEgn3oK8/LLCdXXJ72G2JW2nZaZCeGif2G0tgsk5+5Ekzh6FAI33HADDz/8MADXXXddt+f94x//SOyohjBh+6mjqgo++ijSOETY7cg1azCXLiW0ezcsXEgwEMDp9caGdEqJaGpKqAbaF8eeUVMDTU2Eduwg1PF4B5+APzMTh8cT8QkYra2EhoiG2FHbdufnq3BVre12SbLtTjSJo0chEBYAoBf6eLE8HtKCQUR1tRIAoMwNfj9MmABTphC47DK8BQW4qquxXC5Mu71dEAiBzMhIqAbaF8dePPbf6HP8I0aoyKlgkOCCBTQM1VaUQmhtV5OSxJUsZlkW3//+9wnoPqi94i0ogPJypMOhKpK2laCQGzYghcA3eXLE8estKFClKRYsUOe1xfkHlyxJqAZqeTztCVlhujF1xJPA1fGcYHo6gauu6nN4aLIR1nabpkzpk3NeoxnKxOUYNgwDwzDw+/1DvkREfyMB3+zZpFVXY7cs5M6dyA0boLUV2WFxl0Dj6aeTNWIE9tmzlWCYMKHbBehYa8r0xbEXb/6C1po1muFB3NFBl1xyCXfddReXXXYZeXl5MZFB2jEcS0t+PraJEwmtXYs5dizGiBFYp59O/ezZMYu4r6CAzPXrkatW4W9bnG2XXYYbEB0W17BzNzoxzL5kSVxRRBJonDuXrFGjMCoqsMaOJeB2k7516zEv4D3ZiIdqAbShOu5UQH83/UfcQuD//u//ANi0aVOnvw0Hf0HkJtu7F7fbja+gAGfbTSc9HuXcrarq8QaMvlGDU6ZgHzUKKioIFhfTWFJCRocIHde8eQQ/+KC9Qml2Nua772J79FGCNltMFI+7uhrHX/4SKREhhMBRX4974kSaezEdCYgIm5DDgSM3F/vHH+PPz4+Mo7W4WPVPcLnIPI4SAUO2xICUQ3PcKcCQvaeGCL0KgdbWVp588klOOeUUSktLWbx4MQ6HYyDGNmBE32TCbsceCJA+ezb+TZugqQlHbS1Mm4a/rg7T7++2uUvkRnU4cBQUIDZswMrMxJw0ifzx4wlWVRFyOFSIaCiEePJJ0s4+G/n//h8EAtjOOguxcSOyqAiys2OieBz79rXXCAL1c+dOHPv29SoEoqODzNGj4bnnEFOmkObxgGUh//hHHBddhDh4kJyLLiJ0HCUhkqXEQF81R6OiIinGrelMstxTw5VehcBf/vIXdu7cySmnnMI777xDU1MT3/zmNwdibANGzE1mt2NrboYVK1RRsc2b1eK7dy/msmWEduzo8gaMWWjHj8dYtQoBGCNHwp//rEw9Y8ZgnnVWJFdAWBZiwwZV4x8Qfj/s2oUoLm4fXDiKx26H3FzE5MmIUAhpmshPP1XHe8GoqUGEQtgbGzGamhAnnIBcvVp1DPN6EQsWqAUyFML46CPMo0eRoRCG34+sqFACKs6SEMlQYuCYNMeqKgiFsPl8MZ3ekj3xLRVIhntqONNrdNCGDRv4z//8T6666ir+/d//nQ8++GAgxjWgdLzJhM8Hfj9G24JARgbitNOwuVw4zjqLtNmzSd+yhfSokrrRr2Gmp2PU1yMKChD/3/+HqKpCtLRg2O0Y69bhOPlkcLmQQiDz8yMlnqXNBjk5yC7CM/1jxmBcfDHiuefUjuW55zAuvhj/mDG9zk96PKRVVGC89ho4HIgnnsDwelX0UiAAzz6LKCgAlwsjNxfjs88wDxzA2LMHMzsb0tPbS0L0UtK4u0gkm9M5YGWIj6kcc0EBjtpajE2bENu2YWzahKO2VpkCNYNKX6LbNH0nLnNQbluNe4/HQ0tLS78PaqDpGBsvnU6Ew4HldCLy8jA9Hnj+ecTOnXD0KEybhty6FfeoUZESD9E1d4y8PJVAVVraXu+/tVUtwAcPYlRW4hg1Cqu8nNaXX8Z2xhmYI0eqfILrrkN+/DH4/TFRPO7qaqz33kOMHYsIBpE2G/K99+Cyy7qdV9gk4tq5Ey64APbtg927obgY6usRwSA4nar2z6FDMGIEcs0axNKliHvvhYYGOHQI48Yb8R8+HJf21VUkkmP2bPyPPILZ1KS08ssuI1hS0sn53Ve6M/kck+ZoGPC5z2HMnQvNzUrwGYY6rhlUdNmK/qVXIRAKhdi8eXPkd8uyYn4HOOmkkxI/sgEk5iZDxb07li8ntGkTZmkpPPwwYswYqKtTpqGKCuSiRfDWW5ESD5GaO089pbTr+fPVQm63q5u2rAzxySfI1lbYvx9RWIjvxBOxpaVhrF+PbKsdxMSJcMUVhIqLsXJz26ODamrw5+Zic7lizBWim4Ut2iRiHDgAn32G+Pa3kYWFsH8/UkpwOMBmgzaBZ7a2IqZPx3rhBYxLLoHGRoQQiOefx3H22fjjKAnRMRJJeDz4V6+GpiZ1gsOBY/36Lp3ffREEfe3k1qvmWFuLEQwi7r8ffD5wOjG+/W3E0aMwDFqoDmV0SHL/0qsQyM7O5n//938jv2dkZMT8LoTgj3/8Y/+MbgAJnnACjowMEILgyJE0FhTgPPNMzE8/JThzJmZLi1pMMzJUdI7NhjTNSIkHJ+Dfvx/neechDh5UvYpLS+Gaa+DNN+GTT5Tpp7wcPvoI4feTVl5Oa0kJtkcfRZSWRhZ2XnwR60c/ivE5RDeAx+lUB3tY2KJNIjIjA9PphN/8Bn78Yxg1CvHpp1gZGVhuN2LePOTpp0NuLtaTTyKOHMF66y3Mxkaor4cxYzAqKnDMnh1TSK4rYiKRQiHsTU04Jk3C73KB16sc0489higthZycY3by9eQsPBbNUbS0qB7Mra3qQFMTrFhB2uc+R1Pco9L0F7psRf/RqxBYsWLFQIxj0IjWKK1QCNLTsc2bF7nZZHY2wmYjlJGB4XYrU4FhQH09hsNBMDMTy+PBqKnBUVysFtoZM5BXXYV4/31oaUEuXQqffaZKR7zyilpYDQOzqgrDbifY5gsIY2tuxvz0U4CIxtPXha2TSeToUbUzaWyE999XwiktDfLyaN22jZBhYE6bhrV/P+Yzz2B4vchgEDFjhnqvc8/Fv2kTzurqHhfrjouztNkQL7yA49vfRu7fj3C7ITMTMWMG9lBImdx27eo0397ozuSTtm9fTIiuUVGBVVzcezmL2lq1MwoLAQCHA1lTE8doFEMplr1jSHQyjzXV6e/7Ku48geFKlxrl2rXk+nwEX3wRy+HANnYsxoYNyNpaRFkZTJumtHubDebPV3WAAOH1IkaPVk7WN96AzEy1gObnI+vrYe3aSEE5SksJHTiAWVaGjDJdOA4fhspKQlu3Yn/22RhTSUzCVy8LW7RJRDQ1YQWDiFGjoLgYWV4OW7YgHA5EejqO4mJaCgtBCI7Onk3uv/874q9/xairg+3b1e7lv/4L5yWXYNXVxWhjHW9Qs21xjo6yMXJy4N13sdavR1x4IcasWYT++U9EIIA9FIJ585C1tThXrsQ+YwYNp5/e603elcnHUVtL6MMPMSsqcOTmQlsuBKZJZm8mp7FjsbKzMRyOiJC1XC6s6EitDkTPXRYWYtu2bUjEsncVEp2sY011BiJHIuWFQFcapTlyJOKJJ5Tpx+vFD6TNmqVqAjU1IQIBGDmSYHY2rcXFqlREQQGZJSWICROUKWHVKmXjT0sDpxNr2TLEvn2I1lZkWhry/PMJ7dpFaM4czDYN39bcDJWVMH8+oQMHOpk4os0svS1s0TsHmZ6uzFcXXICsqcEIhRDPP6+09LQ0xJe/HLnOAo6cfz4ejwf55z/D6NHw1ltq9/LYYziiHNFd9uq9/HLM2tpIToNpWZCTg1VSgqytBY8HuWYNhteLCASUc/qttwgB5uuvYzY34y4ro7kXO3zHnZEtGIRp0whVVERMTgSD2NLTlQmtF5NT6KSTEMuXw913qzwOhwOWL6ehpKTL8zvNfeJEzLffVkIHejRzDfaOoWNI9FCIux/sz2ywGIgciZQXAhGN0uFQi0cwiC03l1BWVnsVUK+X0Pbt2EaPxnruOeVUFQKjrAzZtig6q6sJffopYvZsxE9/qpzDQqjom1Wr4IILCF5yCYbXq2r0HzgAfj9Wbi4tkyYpDf+zz+C88/Bv29befMbhIO3gQdL27UNs2RKTbNbTzRDtTLNqanAtW4bx1FOISZOU8zMvT4WkmibW2rU4LrkETjghcm2ooQFj925EU5MSZCNHIjMy1ELeRpc36NatiKlTkXv3tn8G06cTSksjlJOjNH8pERkZSgD4fNDaqnYNgQCsXo3j/PN7FQKdHNCZmfiffx68XgyfL1KVVbQ5eXuNDhIC/9ixOL7/fUR9PTI7G//Ysd0uNB3nbni9qhdEWOi0fR4d3zMZsl+HWty9ALLefRdz40b1/Lhcce8YhzoD8V2lvBDwFhRgv+wy1d3rsccQUiJLSjBOPhkzMxOjvh7L6VQRNePHKxNQIKB+TpsWCSE0amqwAgFkQwMi7PA0DLWwSUlISkJTp7Zn47bZ9H0FBWS++y7Gxo2INkds2qRJWHl5WGlpmI2NWA8+iGhqUrbzBQvaG9P0cjOEnWm2ggLSN21CSKkyjTMyYM4csNuVKWrz5tieBoA1erSaW12dGm9uLpx+OoZp4q6uxltQ0OUNahw9irV/P9ayZaoUhmnC889jjBhBCBAjRyIOHYKGBhWKW1urhMwXvoARCmEJ0WksXdHRAS2CQRxjx+J3uVSkk90OwSAyDic6tGUMr15NaORINW6/Hz7+GNeYMV0K2Y5zt5xOTJutXeh0857JkP061NpFumtr27vvBQKYdnvcO8ahzkB8VyknBLraVgajInTIzMRvmqStX48oK0O+9hp204Srrya4dy8yvLi1afOiqgry87E8HkL19dhPPFEtyjt2gBCIvDzk+PFYeXk0lpTg7BDmFn2Dy0AA0+dTzuesLIxAALFzJ4HiYkynU2nOq1e3Zy7HeTPkbtoEN9+sHMPZ2XD++fD3v2PY7cox++Uv4x8/HlfUNYGsLNLmzEHu3g3p6YjTTkO89Raypga724194UKCJ5yghFn0YuhyIYSIaUrjKCzEcqlXlzU18IUvKP9IKKR2JDNnqrwKy4KyMvxjx/Y6J1d1Naxd2153yemEzZsxp01D7NyJceGFsGULAaczvrjymhocWVnKjBReaBYsINTBBxKZZ4eHM3TgAOaCBch9+9QJ3bxnMmjhHUOikz3u3nHgQGz3vT7sGIc6A5EjkVJCIHorbmtuRgSDuJYswT9qVCRCx3Q6MY8exaqsRJx6Ksbo0cimJvjnPzGnTEFu2YI/XDU1ahH2FhTgPPdcuOsu5KxZSuu225GBAHLWLOQjj5BZXq62/VFfYPQNboRCaifxwguIm25CHj6MVVmJ6fEQdDpxjBsHe/di+HyE+nAzGG2+CAIBFR20ebPSVt1ucLuxtm9vN32FP6uqKnxeL47vfAcjJwf52GNYra0Ir1dp7qtWqZyGDjdoaMYMmDEjZuvuX7KEYFmZypx2uzHWrEGceKK6xuGA9HQ1jlmz8F9xRVwasVFX12nRZsECzEmTkD4fgYICzMmTcQiBf+rUXqODhGki16yJXWjWrMG2aFGX53d6OP1+/HPnErzyyh6T4JJBC482Faa1tBCIIzrIALJ2744JSrB6OD+hBALtSZdhgsG4doxDnYHIkUgpIRDeijsqKyMtEcX27ThvvRV/1IMpfD6w2ZQDt7JSmVBsNowJE2DLFtJMk1BODsHFiyOLsARlPtm3D1lTo8JCN29Wi+vOneByYXviCXIyMmgdNar9i4y+wcNhlZaFJSWhjIwYE4N/xAhs+flY555LcNQoMIy4ykHL4mJwuWDECOUIbNu9yPHjCebltSedtSEAm9OJzeMhZLcrwXT0qOp6FmXvduzbRzA3F8dXv4o8ehRr5MhItVSxdSs0NSEyMmDqVFry85H5+aRXV2P/9FNEWpoKnw0GEfv3w9e+RmDsWBrjbExjs9uhw6JtPPMM1rhxBGpqcIRCancVDGLOnElmL2W3ZSAARUWR+wIhoKiIkM/X9fn08HD2MaN6MLTwsKnQ7fHQ0ksYrAHkr1wJK1aA34/pcJC/fDm1ixcPiCDwjx2LraxM+cyCQbDZsCZNimvHqOmdlBICRk2NisAJP+gAfj9y40Zs8+YRfPFFAGRGhiqqtmOH0uhBafWZmXDiiZCVhczNVX6BKKTdrmrs+HwYO3bAiy8qwTB3Lo6GBoxdu5AjR2IcOoR9yRIa585F5uVhXHyxCiH96CNVqqG0lFAg0KWJIbh4MY3Tp/ep3LM3L4/0G25A3HKLivLxelUHs6ws5cSM0kTDuyW5ahWishKzshLxrW8h3W4oLIw4PcPhmM433miPaGpoIPvCCwmtWhWb+xBl8xZ1dSqJ7umnVYJWWw5CaNs25PPP9x7K2UbI58PecdEeMQJZU4Pj5JMxHngA6XIhW1qUc7s3u3thIf6iItUqM5yRnZ6OVViIu5uolGNJYOpOeADdvs9gk7V7d0QAAOrnihVkzZjB0W6ipxKJt6AA96WXIh96KPLdcOmlSWu+SiQ6RDTBWB6PsqvLqI/PbseSEn9xMdaPfkRaSws+txvb7t3Y9u1TzXNsNsSyZchHHoG6Oqzp0wnV1MCuXbhKSiILi3/0aGwLFigN1DSVtjtqFNJmw/j4Y/Xezc2YO3ZgPPggOaaJtXIl7N+PsW8f8uKLkYcOISdOJLRnTycTQ7ivQfYHH/QpUsgMhQg9/zzm1VerxXLmTOTDDyOKimLrExHruIz0D37vPcRPfkJo507sXi/CbofSUlVLKbwIt/kqjHffRTQ3t0fIQIzNO6zBW5aFYZrKMbxmDeLmm5UfJU4nacjj6bRoW3Y7ZlYWYsMGhJSIadPA4SBUWAiHD/dod7eKi2HhQoKrVilTmWnCZZf1S+x/R+GRDBFDPWFUVLQLgDB+vzo+AELAWV2Nf+NGzIsuavfHbdyIc+7cpA1pTRQ6RDTBeAsKcC1Zgti9G2GzIYRALl5MqLZWhWq2bY+ba2oQBQW4S0pwhaN2fD5VwrmkpNsQwJb8fGxz56oqolJiXHcd8q23EH6/KhN97rnwwguIxkYMu105WisrsQ4dQmRnI95+G/mf/0lLaWmnBjaioCCyUJi1tX2KFAr5fNidTkL/+pdasLOyEEuWEJo9m+DkyRGzkjF+fCTZK0zQ6VSmrNbWdhOPYSCnTlV+hrBADQQiD6jR0X4bVUXU8Hph4kSM9HR1vd8PlZUYO3d2+Zn29F3aOyzatnnzkP/6F2LqVFUg79FHwbIwi4pwLFumEuK6o4tG8wKw3XFHv0fyJEPEUE9YxcWYDkesIHA4ekykSyRGTQ00NcUEG0SOJ8Hn05/oENEEI4Gj559PvmUhXnoJCcjVq3F89audauJEwivdbmzPPos5dqzaFUydir2t4Fro0KEYh54EGk4/HVdJSSSLlIULSd+8WZWfeO89RHOzMiM1NiIsS8XTS6kcyIB8+23k+PE0T5kSMx5XdTVi1SrsR49itLRAXh7Wv/6FuXhxr5FCXWnNwdpagpMnY9u+PbIAifR07BddpJzOUTeeWVwM69djlJWp8Tud8NpriHPPVSasNid4+DMRS5YoU1gXVUQ56SSMceMQjz8OwaAKyf3855FeLzafT5lg4nCSdpVB3WpZOEpLMQ4dQuzerRzfUqoSHxs2wJVX9vqa0Rp6xtatA9JjIBkihnqioaSE/OXL201CvSTSJRrL48HMyMAMh+928ewNV3SIaD+QVl1N60svqeggnw9ZXEzwnXdwnnlmJ60rrKEFbTaCPh/O+fMR996LmZGBcLkwly/vVniIgoL2kgL5+ap0wuHDakdgtyMXLoTPPutsmkpL6/LhN2tqSNu5E7ZsActCSIk5cSJSiF4jhTppzRkZmBdeiLl/P3QwK4VeeKHdP9K2iBuTJmFWV8N99ylHrN2OsXQpVloaoqgI6fMhzzuP0KFDUF5O3dy5OKdNUz4YpxP/I49gq6lBtC2m4p131PvZ7cpn8vHHcOaZylwWp5O0Y54ApknavHkqCS0QgE8+AY9HhemOGNFjxdXukB6P6ioX3dKzrIyWBC8+yRAx1BMWULt4MVkzZgxKdJCvoID06dNjHNNdPXvDER0i2g+Eta6YapzdaF0xjWLy8rCeekqVlM7NxcrJ6VZ4dLTx4vUq2+lttyGPHFE1hV56CS65RNnBW1rUgrhgQbcajgOU0GgL5ZRCIPbtw5g1i0BblFK3kS9EOSPr6kirqCC4ciX2jmYlKaGpidY2/0jYLGKrqYmEYgLq52OPIVasINTSgrDbsSZMIDhzpooCol2jzty6Fefu3YjaWrXIjxyp3mf6dJUwZrcrU1tZGcGrruq90FsbXZlQQi++SNrixYiHHlJj3LdP9Wg4fBjzwgv7vqgahkoIDGc/d0gQTBTJEjEUTVf5NEdLSgbEB9ARZ3U1/nfewTZlSvuOrJtnb7gxbEJEa2pqWLFiBUePHkUIQXl5OZdccglNTU3ceeedVFdXU1BQwPXXX09GRkbC3z/6hrY5nciMjPb69tCt1hWtoRk+H3i9SJ8PK+wX6KYsQPbu3dj++lekzUbQ6VQLXWMj4p57lFbpdCJ/8hNCUiJ+/WvYtg3hcBD0+WDu3C4ffnn0qBIa4aQZux05fz7B1tY+PQh2n0+9hs2mMnaDQYynnyZt0SIVqulwRPwj4Xk5t22DUaPUompZyizm8ahy2Zs3YzmdBPftQ558cqeb03Q6MY4cQezbpxa46dNVxNWHH6qyzYYBEyYg6+oIZGTEfXN3WfPJ61WRSpWVsGiRWlQDAUR2NlZ5edyLavh+Sfv0U0LZ2fCVr0Qyx6MTBBNFX4sD9jdhJcb2xBOIpiZkRgb2XkJs+5NkN5f1N/1dRntAhIBpmnzlK1+htLQUr9fLTTfdxPTp03nllVeYNm0aixcvZuXKlaxcuZKrrroqoe/dUSuXYRt1WxP5nrSuaA3NcjqVc6yoqN0x3EF4RB6etugdIQSOceMQp56KuPtuFYfeljkr7roL43/+B6uuDvnhh6osREYGTJjQafyu6moMUGGaS5aoB7Otx7D1b//Wp8/AHDUK8eGHOMaNIzBiBLbCQlVRtLIStmzBcc01nbbZ1ujRWIAoLUW01f6hqgqysjBzclT009GjWF08lKK1FXHyybB1q7Inh0Jw2mnqs28TZsyZQzAjo0/mmo4mFAHYDx5UceSffqq09p/9DNncDBkZ+E87La4FLKbCZnMzxtatMH8+/ro6taPrBzNNV6ateENl+wN3dTWOv/wl1gxWX4974kSaB2HRHSizXKoyIEIgNzc30qLS5XIxevRojhw5wnvvvcfNN98MwDnnnMPNN9+ccCHQldnA/847iG9/m6DP1+P2KnorFqqrI23ChBhbeUfhEX4vq6REZbAGArB3L2LKFLVw+v1q8RNC1dSvqcFqbSUUHVO/di3Zo0YR9PliyhMHQyHSKiqgtBTrk09U3fs4nXPRn0G4xg1792J3u7GOHEFMmgSnngqTJ6ueAR222Q0lJeRfcw089FB7lu/y5YiHH0bs2qWS6i66CNPtBmJ3XvaGBuSWLXDddapYXHExbN+OOPFE1UvZZlOJeH5/nxbXjiYUu9er8g8efBC5fDli+3a45RaE3Y6cORM5ZQpizJheF9XozyrodOIoKoI1azCXLiW0e3fMd56oypbJFh3k2LdP3beZmSoizjSRe/fi2LdvUITAQJnlkpVh10+gqqqK3bt3M2HCBOrr6yPCIScnh/r6+i6vWbt2LWvXrgXgtttuw9OHxcLYu1fFtdvtMcelYWA76ywA3FKqmOeqKozWVjxFRZHm74AyfQBISdqcOUoLLizEKi4mP+q8yHvV1iIuvbQ9o9XhgBkzYMwYOOUU9drvvovweLAfOYKjtFQJh/x8pW3eeScyJ0fF4r/9topy8fmUPb25GeMXv4DiYkInnUReHA+CsXcvIitLZQxLibjySnj88Yhmy3nnIXfuxGhtxWlZpLW04I7+jKXEKClBXHSRSmZzOFSU0LhxkJeHME3Enj2kBQJ48vMxX3yxPeLoyBHE5z6nBEhNDfzoR8oclZPTHkrq88E99+CePBl39OfeGwsXYpxyClRVIQ4fRvzqV7B/Pxw4oHopn3ACFBbCiBE4X3yRtFNOwRozpsuXstlseDyezvfLuHEIjwdj+nRsX/lK+3cuZcw8w9p7aN682HsnDrq7Rzt9D/1AeN7RmJmZqoHSrl0x/S+MzMw+PXuJwti7V5Ur+fKXI60/OXyY9KYmXG2Vb4+FruaedEiJ+fLL8OGHynfodpM5cyah887r833WHQMqBHw+H7fffjtf//rXcbdpjWGEEMrW3AXl5eWUl5dHfq/pQ7cnt9uNPRDoFHkRcLtpqanpZC5ypqcTmDev+624y6UWP1DdqLp6L58PfD7ML3wBo7UVeckl2HNyEL/9rdLg09KQP/0pocZGjPp65L33KmHx+c8jtm8nMHEiQZ8Pu9eLuXcvNDWppudCwLhx+DIzaSwqgiNH4voM0jMycLtc8Le/qffJy0N87WtY48ZhffwxoYoK8HpxOp34AoHIZxOZV3U19pUrVYiew4EtEIDXXlOfw+uvq8WrvBxZXU3Lp59if/zx9iY5oRDmX/+qqpb+619w8CDC41EOcZ9PLTAnnUSovp7a2tq+azht30eOZWH3+zFdLiV8wmUuTJNAIECwuZnQnj00uVxdvozH46G2poZsy8JWUxPx56hJOAiUldHickW+c3d1NfbHH2+PMnM6CT7+OIFuqo72hNvtxm6zdQqB7Pg99Acej6fT85STlkba9OnK59NWDZfp02lNS+NoP4+nK9xuN/aGBlXRNkzUM3ysdDX3ZCO9thb3unUxvkCOHsU7enSfi+cVFRV1eXzA9lPBYJDbb7+ds846i9mzZwOqf3Fd2xdbV1dHVlZWwt/XW1CgipyZpjrQwYzT7Va8ujryGgL10Gds3Yq7upru5G/Me3m9hHbvJjBrFr6sLOT69cgTT0SedJL6+fzziOpqOHgw9r0PHMBoS8oRI0eqv4c1ZimhslL1C+4LlqXCMMOvU1eH/Ne/8JaVEZo6FVtjI/ajR1WNoq4qX7YVazMfewzx5JOIp55Suxoh1A6mrg5Wr1Zhqx2ceDJsTsjNRZaVIU84Aaqr1c3sdiutbscOzMOHcUd95n2lsaQE45vfRKalqQiWgwehvh750UfYPvsMR12dyrjuDilVuYxHHkFMmoSxdavq8taNz8isqcFRWYmxaRNi2zaMTZtwVFaq+fcRX0EBjunTMR9/HPHkk5iPP45j+nR8g+T0DLW0YB05glywABYuRC5YgHXkCKGWlkEZT2/P8HCm2wqqBw4k7D0GZCcgpeRPf/oTo0ePZsGCBZHjp556Kq+++iqLFy/m1Vdf5bTTTkv8e9NziFVvkQd9aWjR3Xulb91Ka14ejmAQUV2NtCxkfT2Gx4P17LPtDtfCQmR2NkZpKfa0NPWFz58P69erhdxuh/nzCfaxeqKoqcGfn48tPb1z0pPNpuzlXi8yK6tTPSRoK9b22muqnHQopD6v119X23PDUP9GjYKaGlwtLfhNU9n7QZVzaDNFyeJijOZm5Pz5iEceUT4Smw0WLFCF+Y7D8ZhWXY2vogLnTTfB3/6GvPJKxKOPYjgcWAcPIpYu7dGGbFRUtJfLCIUwly7FaG1FLFhAfReROqbTqaKQwnkexyqgSb4QyJDHgz87G1tTU/t4srMJDZLpZCDCJJOWAaigOiBCYNu2bbz22muMHTuWn/zkJwBcccUVLF68mDvvvJN169ZFQkT7g55CrHpL1OlrQ4uOyWLpW7eqsFTTpLWwEFtWlnqwMjKwTjxRLYp+v7qhP/kEc8kS5FNPqdo3Z58NVVUEr7gCEZ0p2eZHiZfwHCO5ES4XZnEx9qNHsTZvJlRRQcjrVQvYtm0x9ZAAQoEAaaedhnjySWXOcjjg4otVWerRo9WOwOMBnw/54IPYv/Md/K+8opyr6ek4vvc9QvX1GCedhBw1SkUKff7z7eWr33kH8bnPYR6HjdOoqcG2f7/yO2zcCHv3Ir/7XbXwO51YptlzaGdVVfs94PVGShSEfL4uF5tgIID9C19AVFQg/H6kw6ESD4/h4exL7spA0FVJjmPRvBPp0OzvMMlkJVxBNRwZhRBx99yIlwERApMnT+axxx7r8m+/+MUvBmII3dJlos5FF0Vu+JjtWGYmYto0xOHDuPbvjyRGdaRTWGpGhgpLfecd9aCnp8PChTRNnkxeVDq+mDgR+d57BKdPV4t+IIAxdSp8+imBtlaQx/IwxszR4VBN2N9+G/n++5g7d8Ymi3Wx+Jjhss1NTZESF+LFF+EHP1BNapxO5Lx5qgRGbS2hrCwCbclm0uNB7N5N2gsvwNGjKtdg3jy4916V/dw2J1FQ0F6m+hgIFweUhoEoKICpU1VYblqaynD+4Q9VGY/uKCzs1CCnp3BQ2dbAXrSNWbQJenkM+QPJljGcCM072YviDRVaCgqwXX11TM5GcMmShO4QUy5juCMdb3g5fjwNLlf7jRrejmVmYkyf3t4ApraWrK9/vcubupOfoakJ/6ZNXYalRqfjGzYbwX/+U72e06k05bo6jK9/nZDdfszaVEwTkYMHsR58kGB+vqqJE9WtjIqKLhcfo64OMjORdXUqWQyQ6ekqgunzn1cL7cqViJNPRjocWCNHRrS29Opq0u66S2nnlgUbNyIvu0zVF7rgAmXi2rcP2dCAcRxb3HBxQPnHPyKvvBLj9tuVcAoEYPx45FtvKbtyGx21VOuEEzo1yOlN4Bq1tYh//CNST8fopTZRT2NPtozh49W8ky3sdagybDKGk53oG97p8agWiG2Et2NixIj2HUFWFiGns9ubuks/Q1MTQZ+Ppg6F4SyIpOO7q6tVK8voa/1+WkeNOu4HJzxHo6ZG5QlAp25lHXdBkWttNiwpI74LKQQcPoxsalJ1gNo0epmW1il3wbVjR7sAAPVz717YuhUZ7tAmJbz4IjIqAuxY5ld3/vnkpKcjtm/HmjZNFahzuwnk5RFMS4sko3WlpZpLl3Jk7txO7T+7e9jSDhyAV19Vgiy8cL/6KmmXXNLnqI1ktHkfbyexRGf59nesfCqjhUAveAsKcF55Jbb330e43UrT7aFsBLRv72PCB6OqY3Z3Qw+ERtjR9BDdrcyYODF2F9SGf/RobBddhGzrg4zdjrFkidrVjB+PfcQIxHnn4Z81i4Zx42IWCwFgt6tSEZalkuJ8PlVSu6lJOaYXLEB6PASOsy6NBdTNnk12YSG2DRsQJSUqOsnphKiaTN1qqXEkk4UxWlsR06fDO+9EQvfE7NkYra3HNPZksnknopNYIk1cqWxa0k1lBplwOn/wjTcwpk3DGDECmZtLIGxb7uam9hUUkD57drut3+HA0Vb1sLdoo/7WCLsSNMHFi2mYPp38DrugMDF9EsJ9g2fMUH2DTzsNfw/jDBQV4Vy0CP7+d2U2OftsxD//iZw8WVX6DIXg5ZeRt9+esNLEgZwcHKedhrjvPqxQqFPVSaOmBhEKYW9sRDQ3I9PTIS0N10cfEXj55fgeNqcTsXGjSkoLJ1Q5nZDgjPfBIBGdxBKp0KSyaUk3lRlkYrpsbdiA47zzVAmBrCyCbc7drm5qZ3U1/k2bVJhhOKpn+3aydu9GtLRgf+MN5LPPdhlt1N8a4bEImo59EmKu6cX0EcjIIC03V0UTBYNQVga7dhEoK8MoKIgswr62+kTHQ1hrMrdsQT75JGRkYGRkEPR4YkIupcejSnC0mamEYWDMmEFgypT4H7amJmRNDWLs2IgQkG3NT4Y6iegklsiieKlcQM6oqQGHA3P06Pa15MAB3VRmoIi5+bxe/IC5dClMmUJg8uRuF89OnZBcLhy5ucg//AHj3HMRDz6IcDqxXK725I/zz++zLflYORZBc6zCSVRV4aurwzFpkio1MWaM8i+0tODPyVFltU0zITHoYaFtjBql8g+mTgXLwhg7VpUEaXtw7C0tqs7Q5s1qATdNGD8eMzs7pnNVTwuNlZWlIoq2b2/XdCdOVMeHOInoJJbIonip3FRGFhaqaL62Uu6m3Y65YEHPXfL6SGpUYDpGLI+nPUsRIlnAvsmTlUYZ53Xm6NGwZg3SZkOGK2k2NmK0CRijtRXbkSM9ZiMnCx2zpw16zqYOfxahts5jwZYWWLBAlfOGhPo9wkLbys7GmDEDsWYNPPMMxj/+oYRw24Nj7NuHtWEDcv58lRE7fz5s3Kgcx9GYJrKt0XzH+VkZGaowXVupcOl0ImfOxOqHUugDTUNJCSxfrvJBIFIwMJCT02vWfJh4MvHjJdkyqgeUjtn+waD63TrefXM7eifQA8dq1+x4ndHaGilBbQYCGKWlqjhXKKTCIgsLCe3ejf2FF5La4dWVkyo9qix3V3b0rrpCcf31+L74RVVFNYF+j4gz0rJg06b2BUjKmAcnoulGId1u5JQpqgx1+LvuodF8uLSCWLhQ5SfYbMhBLK2QSCzgyOLF5JxwghKYY8cSDASw/f73cTsnE2nCSbaM6oGku2z/vnbJ6wktBHrgWB21Ha8TTieyslJ1v9qzB/Pzn0e8/DIiKwvZ0KBaM4aLdSWxw6ujdmdrboYVK1SZ5R07uhx/lw/wli04S0qQNTVYTie+BAmBsPA1PvgAq7UVY+pUhGEQys0l0Fa6m4ICGktK8CxdCr/7HbS2ItLS4MYbqTv9dBxRfo+eGs2HPB4C2dnY6+qgza8RyMkZtNIKiUQAGevXwxNPIJuaMNLScJxwAv6oNqS93aeJjA5KtozqgaRTtj/oHsMDzbHawqOvE0BWeGfg9eKvq8O88UblD/jkE0IHDqiHC8DhIO3gwaSMh+6o3QmfTzkMfb52W3qHh7PTA5yXhzM7G665BguOKfywO8LCN3vUKNLefRf27CFkGHDoEPayMgJtD05adTWtmzZhnzMn4pg2PvsMR3V1l43mY2ibX8uUKZEIMNmWLGZfvpyGYbAodWwqYwQCsGcP5uWXE9q6VZ3UyyKcyOigZMuoHkh0j+FhQlc7ioa22kL2NWvab+42B7L14IOYNlvSxUN3fBil04lwOLCiyz10eDg75kwYp50Gt94KGRnqhj6G8MNeaSt9TEWFMgF1aEJi1NQgQyH8mZnKMQ04u1jUelp8nNXV+Ldvx3HttYj6emR2Nv7t23GGBckQxrFvX3utGlCf265dmHZ7u7DvZRFOZLhzMmZUDxQSaJ47l9yMjIhprm76dJ0xPBTpakfR8eY2i4vh7bcJhqOEksw81HG8wfR0HMuXE9q0SZ3QxcPZKWdi2jRlool2uPcx/LA7ott7Wq++ipgzByMzk0B+fkxv4Hg1y54Wn/TPPsNht6udQFuymGPBAkJ1dUPfRGG3K8HZVsbDMk2MnBxlNoMBX4STMaN6oDCB/H/8A/F//6d6lDid5H/zm9R88YuEer06PrQQGEQ63txGSwv+7dtjT0oi22dXD2NjQQHOM8/s9uEM50w4Fi9WIaKTJiEBIxTCCguCPoYfdkdMe0+fD/nmm0ghVGZvVMZ2b5pldEZ3cOJEuOEGRFVVzPxs4aJ60XXe16zBtmjRcc9jsPGPHo1twQKMp5+OdPKyli4lcPrphE46Ka5FONGZrsmUUT2Q5Hz6KeKhh6C5We3MmpsRDz1EzowZ1E6enJD30EJgkIm+ubusHZRkts+OD2NvoYJmTQ2OnTth3TqklMiPPsL81reQ//iHOqEt/DAR2cJh/0PowAHMBQsitZ5EMBizyHeVyBSaNQt55EinxSssIDouXiGfD3tRkaqDFC7xW1REyOc77nn0RKJq6EReZ+9e3G53zOt48/Nxn3yy2pnV1UFuLmRndyoJ0hOu6mpYuxazpKQ9tn/t2qTZ1Q4VbBUVqrzK/v3tmenjxmGrqAAtBIYfQ832GY+2Zzqd0NiImDMn0rQ89PHHcM89hOrrj6k4WXdEzDzhxL5lyzBaWwkuWBCTrdpVIpO5dCli5sy40/RDHg/+oiJs2dkx9aH6MzooUdp19OsIux17IBDzOq66Oozt2xH33x/ZCRjf/jauGTNojrOXRbgbXcckp2FhLhtIsrJU86LoAoyVlep4gtDJYklE2NwS+NGPCH3jGwR+9KOkcQp3RTwJQUHLwli8GMPvR3i9GH4/xpw5eNPTOXLWWRxNkACALtoQCgEnnECgQwJXT+PuKr7d1tyM89NPY5Kkwu8VbAsN7amMSKJIVAJWb6/j2rtX+TqampDBILKpCVasUMfjpFtzmd3ep7GmOoG0NOTll6te2qYJLhfy8ssJhP0zCUDvBAaZrrb3Q8X2GU9CkM1mQ+zejVi7NlJ3n6wsbOeck/DxhIWo+4QTcL7zDqxdS2D7dmwvv0xWlKbb07g7Oo0dhw9DZSWhrVuxP/tsjMbcNHcuObm5kaiNo1Om9KvATlQCVq8tVY8cUfkTUkZMXbK1VR2Pk8Eylw03WouKVC7K5z8f2UnLujpau2kafyxoITCIDPUSufFE2dhrauCxx5DNzeqA1wuPPYZ9wQIYNy7hY5KAtCyC77+POXky9i7s0T2NO9okZ2tuVlvv+fNVLkeUachXUEDe008jHnoI4fMhnE7yvvY1ahctStjOpiOJipfv7XUCJSU4Cwray38bBgiBVVRExtatcfkiBsNcNiwxDKwZMxAvvYRobUWmpSEvuKDHftl9RQuBQWSol8iNy4fR2KhaPmZlISwLaRhIKVV/4n7CqKvDUViI2LcP4fdjOByYxcURe3RP446OgDI//ZTQ1q2xyXxtGnNWUxPGn/+syn+0VSIVf/4zWdOnJy7fIXpOgKOlBceZZ8KePViWheVyEZoxIy4TVPSOU3o8cNll8M9/qj92+N4CeXk4v/lNuO22iE+AG29E7tyJ+eqrcSkriepTnOqIqir81dU4yssjmen+qqqe+2X3ES0EBpGhXiI3nvjtQEkJzgkTEJs2qU5fNhtMn37cDWR6wpaWpvIOnnlGRQfZ7YhLL8XWZkftatz2yZORtbWx83O7EaJD/FObxmzfsiUiAAD1c9cu7Hv3Hne+Q0ciTV4efljlWbz9NkZ2NoHiYlUptRe6i3gK3nADsqmJQIfoINfu3chHHoHLL1cFy2w2+POfsf/85yo2PQ5lJZVj+xOJ9HhwVFTAq68q5UkIHGVltOiyEcOD4ZAO31v8diAjA+e556ribGGfwIgRnZy1icSor4f334ezz263R7//PsayZd2O29222EcvmFYohKO2FtvUqVgNDUghCC5ejLegAHdmZmzHNMNACoFsy0BOJOEmL+LUU+Gll5AnnwyAOX48wTjCLrvbccqJE7FmzaKlQ/VUceRIbBlpKZVjt76+/dgQUlaGNIYB06Zh7NoFra0qZyMq+z0RaCEwiAy1kNBjwVFRoZyK+/aphisZGcicHBwVFf3WP0FKCTNnIlatiggeuXChOt4LnRZMy4I33sD40pcI1tcrrRjwjh9P5pIl8PDDkfcQX/0q3vHjEz6fcJMX4XAgzjlHxe6HQoiKChylpb2GXfbYmKQLAmVlOM87D/HXv0ZaZ3LllYTy8tpP6kVZGer+rmTBqKpS5U+uvBJaWsDthk8/Vce1OWjokwpbZtMwMN54o73ZvGEgGhowL764395TZmXBa68hg0GlMQWD6vd/+7der4020dl8Pti9WyW5HTpEqKICdu3CVVIChkHI48FYskRpaGlp/baDi5S+LiuDDz9EPP88BALI7GyMxYsjZq7u6HNjEsuCDz5Qn53Npn5+8AHMnav+HoeyMtT9XcmC6XZjOJ1w990RgWxceimm252w99BCYJAZ7unwxtGjsHVrJMIEIWDrVnW8nwj5fFhlZRhCRGzaVmlpXOGJ0SY64fMpU0hbQxz14qGIBu0/eBBz/PgY7TqRDrswDSUl5C9fDnV1iJdfVnNKS0O29WeONnN1PSkLtmxBuN3tvQ+2bOm2MYlRUYFsbIQpUyKfn6yvB6eT0De+EZeyMhBtEVMB0dSkFKhoYbpxozqeILQQ0PQr0mZT2mRra7t93ulUx/uJ4wlPjDbRhaukRkJEIcYMYvr97S1EO/wtkVhA7eLFeF5/HelygdutFmDTxLLZVAG7HjBqapT/4sQTI45y2dDQrTlIFBQgnE4ICwrDQEyYgDVqFE1xliqQhYU4Ro3qFKGVyLaIKYHPh+X1IkpL2/MEvF4VtZUgtBDQ9CtWfj5y/nzE00+32+fnz8fqx37KxxOeGG2is2pqsDc0EHrhBRUi2uF1BtKfYwHNkyfjnjq1vcyzEFBWhn/s2B6vNd1ujLy8SLRU2KTgME2MDz7oVDvIApgzB/bsUd+ZzQZz5vQ5/8E4ejQ2QuvLX+7rtFMe/9ix2MaNQ+7cGYkOiuc77wtaCGj6lVbTxDZtmrLPh52M06bRGl1KOsEcr68l2kRnAFmFhZFCc9E1iAban9NSUIDt6quxPfEEoqkJmZFBcMmSXm3soqlJCY5zz40ILPHZZ5hbtiA++qhT7SCamwlt2oRx1VURwW199BFcemncY3UcOIC1di1GVlbkPa21a3Fcckm/BQQMR471O+8LWgho+hUrKwtfVRWOc85pb76yYwdWAgpgJaqiZk+v37HQXGbUYtlf/pzu5nXMwi0YhNJSxAsvKLOcwwHnnYcMBFQtpA5OW//Ysdjy8uDFF9vzBCZN6pv22Waikrm5ETMGXm97LSFNXHRV8bYxShFJBFoIaPoVb0EB9jPPxJ9gs0lPIYj08Le+PDyDEeHSW2jlsQgemZWlko3aSncIvx9eeQVx8cUwciSmzRbjtPUWFOC+9FJkW0kM6XTCpZf26Tvzjx2Lo7AwEhUmDAMxY0ZCzRipQG+KSCLQQkDTr/RXGGxPCzSQkMV7MDK6+0PwdIyWkoaB8HgQr76KsXYt9rQ0zEWLIk5bZ3U1/o0bMS+6qD2yZ+NGnHPnxj8Gw8A6/XSMgwfbG9OcfnpCk5xSgYFQRLQQ0PQ7/WE26XGBbvt/l3/rwxgGI6O7PwRPx2gpYZqYmzfD0aPqnxAY776rEpLCY2hqio18Ch+Pcwyiqgp/XR3mFVf0ewjtcGYgFBEtljVDEsvjae8bEKZtge7pb32hU3+CAcjo7m3sAnBXV5OxdWukv4EB5OzeTd7rr5Oze3enh7pj7wMRCiHPPhtZXY0sLUWWlmJVVakG83GMIe55+P2IzZth82b10+8fUiVRkoFE3cs9oXcCmiFJbyU3EhG+ORgZ3T3Nq0t/wdKlavFesQL8fkyHg/zly6ldvDgS0tlpHq2tiF/9Co4cAdNUSWd2u/rXyxjixVdQQPrs2ZFxCYcDx/LlNOpEsT4xEKVlhIynoEqSUVlZ2W+v7fF4qOkmiWa4M9Tm3lN0UF8jh5Jp7t2N3V1djf2OO7A1N0cctsZ552H88Y8q6icciulyEbj//m5LWqfX1uL++98RL72EEQhg2e3ICy6g5YorIuGbxxt55a6uxv6//4sjPx8RLoFcW0vgu99NmrIRyfSd90SiouCKumlEo3cCmiHLcC250d28zJoaHJWVqqJkOHRzxgzE0aPIlpbIeUZmpio6140Q8Obn4z7hBITbDc3NiPR05JgxeKNs9cf72Zo1NTh27oR169pLII8bh6XLRvSZ/r7PB0QI3HvvvXz44YdkZ2dz++23A9DU1MSdd95JdXU1BQUFXH/99WT0Y3lhTeowXCtYmk4nxs6dEN33IC+vc+y934/owWbsqq1VJSGefjrSrYpFi3DNnJmwRC7T6VRd2cKGBimhslId1yQVA+IYPvfcc/nZz34Wc2zlypVMmzaNu+++m2nTprFy5cqBGIomBUhUQ/ZkQzQ0qB4J4Wbtdrvq0HbNNRCuJJqWBtdeS6vD0e3rOA4cUHkDZ5yhHMRnnAGvvqqOJ4hgIADz58eOdf58dVyTVAzITuDEE0+kqqoq5th7773HzTffDMA555zDzTffzFVXXTUQw9EMc4Z6x7ZuEQJr61bE/PntWbj79iGFgD/8AdHWOrL10097zsiWEmPKFFi9GtHW7U0sWNCutScAKzcXf0MDjssui/EJWLm5CXsPTWIYNJ9AfX09uW03RE5ODvXRXYs6sHbtWtauXQvAbbfdhqcfw8xsNlu/vn4yM1zmbowfj0hP7xTfL8ePx9nN/IbC3I1JkzAuuEDtctqKsnHllZCRAbffHonCcV17LfYTTsDdTWKW6fGoHg+hkMoRCIXgjTdwLF+euM8gLw/bySfDvfdG6g/1Nq6BZih85wNBUjiGhRCde7lGUV5eTnl5eeT3/vToD5WIgf5guMxduFxkzZsXG1Z30UU0uFzIbuY3FOYunE6yTj8dMy0Nw+vFcrlgyhSsp57CnDSpvWz2W28RmDOn2yiczIYGnCUlGFJihEJYpolVUoKvoYHGBH0G7upq7Bs2YF5+eXuy2IYNBLZt09FBg0TSRQdlZ2dTV1dHbm4udXV1ZCWgoJhGA8O3Y5sEGk4/HVdJSWReZk2N0uSj6cX0FfJ4CBQVYXe7EW3CJJCTE1e/hXhJRNaxZmAYNCFw6qmn8uqrr7J48WJeffVVTjvttMEaimYYkirho+mAs7Y20mNACIGjrIyWHhb0mESutp2SfflyGhL4WVkeD2ZGBubIke07gUOHdMZwEjIgQuCuu+7ik08+obGxkWuuuYZly5axePFi7rzzTtatWxcJEdVoNH3EMOCUUxBHjkTCPeUpp3Qq1GYCOVu3YuzbB6NG4a+uxly6FCMYJGSzEdq0CeeZZ3ZrqulrwpKvoID0U09FvPUWorUVIy0Nc84cnTGchAyIEPjhD3/Y5fFf/OIXA/H2Gk2f6O8+BYlE1NaqpLGpUyMOWGw2dbwt5t8EPH//O/zud9DairDbcf3wh3gbGrAfORLpvdydqeZY8i5ctbXw2WeIN94Anw/hdCLz83HNnq2byiQZSeEY1miShaGWaGaz2+HJJ5F+f/v4HA5sF14YOSdn69aIAACUsPj970m75x54/XV1rIeiZMdSzthx4ACsXo1lWUowWRasXo3j/PO1EEgykiNWS6NJEoZaolnI54OiItV7FtTPoqKIdg8oE1BYAIDKKwiF1G4Bei1K1mvZ7q4IBFRpi2jCLUY1SYXeCWg0UQy1RLOYSJ+2pKyOkT7W2LEYaWmxgiA9HTl5MmL8eMSIETG9kztyLH0V/GPHYisrizis+6NBuiYx6J2ARhPFQNRvTyS+ggLss2djbNuG2L4dY9s27LNn44sSWEenTIEbb0RkZqrs4MxMxA03EHrpJcTDD2Pecw+Z69fTXabOsfRVaCkowH/11VgzZyJPOAFr5kz8V1+dNDkCmnb0TkCjiSJcv932xBOIpiZkRgbBxYv7tZHM8eCqrlaVOs84AxEMIm02WLcO15ln0tw2ZgvwTZ2K85ZboLoaPB44fBjjlVeguRnDsnDU1+OeODFyTTTHkncxXHM1hiNaCGg0HbHZkFOmILxepMulom+SFMe+fbBpE1LK9gVWCBz79kUWdHdtLY5167BWr4ZAADMUgtmzEaedBq+8osw1O3fGXNORY2pwfwzXaAae5L27NZpBwFVdDf/8J6FQqD3TddcuXCUlyWnKsNuVkIp2uNps7dU7aY/UiZwTDMKqVYgbbuj2Gk3qoH0CGk0UxxQJM4j4R4+GBQtiSzYvWKCOh2mL1DFCIQy/H2kY4Ha3C4WurtGkDHonoNFEcSyRMINJS34+trlzMdPTI0XlQjNm0BIVi+8fOxZHURFs3Kji9Q0DTjoJq6wMY8kSQnZ7p2s0qYMWAhpNFAPR2DuRdFVUrpMD1jCw5s5FeL2R0hLWWWcRmDIFx5Qp+Nxu7bRNYbQQ0GiiGIpRLb05YEVVFf6mJhxf/SrU1yOzs/Fv20aopQXbWWfRkqSmLs3AoIWARtOB4RbVIgsLcTidsGIFMhAAux3HggW0FBYO9tA0SYB2DGs0wx3Lgo8/bi/jEAyq3y1rcMelSQr0TkCjGeaImhoCxcU4ZsyIJMD5a2sR2gykQQsBjWbYIwsLSXO54M9/jvQmTvvylwloc5AGbQ7SaIY99oYGePPN2Mqob76pjmtSHr0T0GiGOUZFBVZLC6K0FBEKIU0T2dKCUVEBuq1ryqOFgEYzzLGKizENA9nQENN4xiou1qYAjb4HNJrhTkNJCSxfrjp8gfq5fLk6rkl59E5AoxnmWEDt4sVkzZihTEPFxTSUlKADRDWghYBGkxJYwNGSEtDav6YD2hyk0Wg0KYwWAhqNRpPCaCGg0Wg0KYwWAhqNRpPCaCGg0Wg0KYyQUiZzqXSNRqPR9CN6J9CBm266abCHMGjouaceqTpvSO25R6OFgEaj0aQwWghoNBpNCqOFQAfKy8sHewiDhp576pGq84bUnns02jGs0Wg0KYzeCWg0Gk0Ko4WARqPRpDApW0V0w4YNPPDAA1iWxQUXXMDixYu7PO/tt9/mjjvu4Le//S1lZWUDO8h+ore5v/LKKzzyyCPk5eUBcNFFF3HBBRcMwkgTSzzf+fr163n88ccRQjBu3Dh+8IMfDPxA+4He5v7ggw+yZcsWAPx+P/X19Tz44IMDP9B+oLe519TUsGLFCpqbm7Esiy9/+cvMnDlzcAY7GMgUJBQKye9973vy0KFDMhAIyBtuuEHu37+/03ktLS3yF7/4hfzZz34md+zYMQgjTTzxzP3ll1+Wf/7znwdphP1DPPOurKyUP/nJT2RjY6OUUsqjR48OxlATTrz3e5hnn31WrlixYgBH2H/EM/c//elP8oUXXpBSSrl//3557bXXDsZQB42UNAft2LGDkSNHMmLECGw2G3PnzuW9997rdN4//vEPFi1ahN1uH4RR9g/xzn24Ec+8X3rpJS688EIyMjIAyM7OHoyhJpy+fudvvvkmn/vc5wZwhP1HPHMXQtDS0gJAS0sLubm5gzHUQSMlzUFHjhwhPz8/8nt+fj7bt2+POWfXrl3U1NQwc+ZMnnnmmYEeYr8Rz9wB3nnnHbZu3cqoUaP42te+hsfjGchhJpx45l1ZWQnAz3/+cyzLYunSpZx88skDOcx+Id7vHKC6upqqqipOOumkgRpevxLP3JcuXcott9zC888/T2trKz//+c8HepiDSkruBHrDsiwefvhhvvrVrw72UAaFWbNmsWLFCv77v/+b6dOns2LFisEe0oBgWRYHDx7kl7/8JT/4wQ+47777aG5uHuxhDShvvvkmZ5xxBoaROkvDm2++ybnnnsuf/vQn/v3f/5177rkHy0qd5pup801HkZeXR21tbeT32traiBMUwOfzsX//fn71q1+xfPlytm/fzu9//3t27tw5GMNNKL3NHSAzMzNiArvgggvYtWvXgI6xP4hn3nl5eZx66qnYbDYKCwsZNWoUBw8eHOihJpx45h5m/fr1nHnmmQM1tH4nnrmvW7eOOXPmADBp0iQCgQCNjY0DOs7BJCWFQFlZGQcPHqSqqopgMMj69es59dRTI393u9385S9/YcWKFaxYsYKJEyfy05/+dFhEB/U2d4C6urrI/99//32Ki4sHepgJJ555n3766ZEImYaGBg4ePMiIESMGY7gJJZ65Axw4cIDm5mYmTZo0CKPsH+KZu8fjYfPmzQBUVFQQCATIysoajOEOCinpEzBNk29+85vceuutWJbFeeedx5gxY/jHP/5BWVlZlw/IcCGeuT/33HO8//77mKZJRkYG11577WAP+7iJZ94zZsxg48aNXH/99RiGwVVXXUVmZuZgD/24ifd+f/PNN5k7dy5CiEEeceKIZ+5f/epXue+++1izZg0A11577bD6DHpDl43QaDSaFCYlzUEajUajUWghoNFoNCmMFgIajUaTwmghoNFoNCmMFgIajUaTwmghoNFoNCmMFgIaTTcsX76cb33rW/h8vsixl156iZtvvhkAKSXPPPMM1113HVdeeSXf/e53+dvf/kYgEADgueee48c//jHBYDBy/Zo1a/jpT39KKBQa0LloNN2hhYBG0wOWZfHss892+bcHHniAtWvX8r3vfY+HH36Yn/3sZ3z88cfceeedAFx44YW43W6eeuopAA4fPsxjjz3GNddcg2maAzYHjaYntBDQaHrg0ksvZdWqVZ0KyR08eJAXXniB6667jkmTJmGaJmPGjOHHP/4xGzZsYPPmzRiGwXe/+13WrFnDvn37uO+++7jwwgspLS0dpNloNJ3RQkCj6YHS0lKmTp3KqlWrYo5//PHH5OfnM2HChJjjHo+HiRMnsmnTJgCKiopYvHgxv/rVr6itrWXJkiUDNnaNJh60ENBoemHZsmU899xzNDQ0RI41NjZ223wkNzc35twpU6bQ2NjIGWecgcPh6PfxajR9QQsBjaYXxo4dy6xZs1i5cmXkWGZmZky11Wjq6uoiVSiDwSD3338/F110Ec8//zyHDx8eiCFrNHGjhYBGEwfLli3jpZde4siRIwCcdNJJ1NbWsmPHjpjzampq2L59e6Qz1xNPPEF2djbf+MY3mDdvHvfff/+Aj12j6QktBDSaOBg5ciRz5szhueeeA5Stf968edx999189tlnWJbF/v37uf3225k2bRrTp09nz549PPfcc3znO99BCMGyZcuorq7m5ZdfHuTZaDTtpGQ/AY3mWFiyZAmvv/565PdvfvObPPPMM9xzzz0cOXKErKwszjzzTJYtW4ZlWfzpT3/i8ssvZ+TIkQA4HA6+853vcMcdd3DKKaeQk5MzSDPRaNrR/QQ0Go0mhdHmII1Go0lhtBDQaDSaFEYLAY1Go0lhtBDQaDSaFEYLAY1Go0lhtBDQaDSaFEYLAY1Go0lhtBDQaDSaFOb/B3d4YRt/MtVTAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.scatterplot(boston_data['NOX'],boston_data['Price'],color=\"r\",alpha=0.6)\n",
    "plt.title(\"Price~NOX\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们可以看到，数据给定任务所需要的因变量，因变量为波士顿房价Price是一个连续型变量，所以这是一个回归的例子。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "各个特征的相关解释：\n",
    "   - CRIM：各城镇的人均犯罪率\n",
    "   - ZN：规划地段超过25,000平方英尺的住宅用地比例\n",
    "   - INDUS：城镇非零售商业用地比例\n",
    "   - CHAS：是否在查尔斯河边(=1是)\n",
    "   - NOX：一氧化氮浓度(/千万分之一)\n",
    "   - RM：每个住宅的平均房间数\n",
    "   - AGE：1940年以前建造的自住房屋的比例\n",
    "   - DIS：到波士顿五个就业中心的加权距离\n",
    "   - RAD：放射状公路的可达性指数\n",
    "   - TAX：全部价值的房产税率(每1万美元)\n",
    "   - PTRATIO：按城镇分配的学生与教师比例\n",
    "   - B：1000(Bk - 0.63)^2其中Bk是每个城镇的黑人比例\n",
    "   - LSTAT：较低地位人口\n",
    "   - Price：房价"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1.2 分类"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们来看看一个分类的例子，我们来看看大名鼎鼎的iris数据集："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-14T11:33:18.639401Z",
     "start_time": "2021-03-14T11:33:18.624401Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sepal length (cm)</th>\n",
       "      <th>sepal width (cm)</th>\n",
       "      <th>petal length (cm)</th>\n",
       "      <th>petal width (cm)</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5.1</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4.9</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.7</td>\n",
       "      <td>3.2</td>\n",
       "      <td>1.3</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4.6</td>\n",
       "      <td>3.1</td>\n",
       "      <td>1.5</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5.0</td>\n",
       "      <td>3.6</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   sepal length (cm)  sepal width (cm)  petal length (cm)  petal width (cm)  \\\n",
       "0                5.1               3.5                1.4               0.2   \n",
       "1                4.9               3.0                1.4               0.2   \n",
       "2                4.7               3.2                1.3               0.2   \n",
       "3                4.6               3.1                1.5               0.2   \n",
       "4                5.0               3.6                1.4               0.2   \n",
       "\n",
       "   target  \n",
       "0       0  \n",
       "1       0  \n",
       "2       0  \n",
       "3       0  \n",
       "4       0  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn import datasets\n",
    "iris = datasets.load_iris()\n",
    "X = iris.data\n",
    "y = iris.target\n",
    "features = iris.feature_names\n",
    "iris_data = pd.DataFrame(X,columns=features)\n",
    "iris_data['target'] = y\n",
    "iris_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-14T11:34:18.931215Z",
     "start_time": "2021-03-14T11:34:18.667213Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEMCAYAAADAqxFbAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA6rklEQVR4nO3de3gTdboH8O/k1qQXeguVUq4tLQIKaJE7rEIVF0V0jyCCLCqsHquwrCsP6D4r+hwFFPAgUhYvKF7YFT3ugnBwq1WxR1ZYCygWkJabCAVKm5aWNmmSmTl/xIRMmza/tJPJTPp+nsdHkpnMvJm2eTPz+73zcqIoiiCEEEJ+oYt0AIQQQtSFEgMhhBAJSgyEEEIkKDEQQgiRoMRACCFEghIDIYQQCYOSOxMEAUuWLEFKSgqWLFkiWbZr1y68++67SElJAQDceuutmDhxopLhEUIIgcKJYefOncjIyIDdbg+4fPTo0Zg7d66SIRFCCGlGsUtJ1dXV2L9/P50FEEKIyil2xrBp0ybcd999rZ4tAMDevXtx5MgRpKenY86cObBarUG3W1FRIWeYIbFaraiqqorY/kOhlVgpTnlpJU5AO7FGQ5zdu3dv87WKJIZ9+/YhMTERmZmZOHToUMB1cnNzMWbMGBiNRnz22WcoKCjA0qVLW6xXVFSEoqIiAMCKFSuYkke4GAyGiO4/FFqJleKUl1biBLQTa2eIk1PiXkl//etfUVxcDL1eD6fTCbvdjuHDh2PBggUB1xcEAQ888ADefvvtoNumMwY2WomV4pSXVuIEtBNrNMSpijOGmTNnYubMmQCAQ4cOYfv27S2SQk1NDZKTkwEAJSUl6NGjhxKhEUIIaUbRWUnNbdmyBVlZWRg2bBg++eQTlJSUQK/XIz4+Hvn5+ZEMjRBC2iSKIhwOBwRBAMdxkQ6nhQsXLsDpdMJsNoccnyKXksKJLiWx0UqsFKe8tBInoJ1YvXHa7XYYjUYYDBH9ft0qg8EAh8MBl8sFi8UiWaaKS0mEhJvw8V8hVp5r8TyXlg7dHTMjEBGJdoIgqDYpeBkMBjQ1NYX+ujDEQojixMpzwPmzLZ+PQCykc1Dj5aNA2hMn3SuJEEKIBCUGQgjRqC+//BLjxo3DmDFjsG7dOtm2S4mBEEI0iOd5/OlPf8J7772HL7/8Elu3bkVZWZks26YxBkIICbNwTI44cOAA+vTpg969ewMApk6disLCQuTk5HQoVoASA4kSXFp6wIFmLi1d8VgIaS4ckyPOnz8vmXaanp6OAwcOdGCLV1BiIFGBpqQSIh8aYyCEEA3q1q2bpMD33Llz6NatmyzbpsRACCEaNHToUJw8eRKnT5+G0+nEtm3bcMstt8iybbqURAghGmQwGPDcc89h5syZEAQB99xzD/r37y/PtmXZCiGEkFaFa3LExIkTw9IVkxIDIYSEmdYmR9AYAyGEEAlKDIQQQiQoMRBCCJGgxEAIIUSCBp9JxPnfR6bWHAPe4WksQk12CIkMSgwk4vzvI8MbDIDb7Xk+kkERogGPP/44ioqKYLVa8cUXX8i2XbqURAghChBFsc3H7TF9+nRs3ry5w9tpjhIDIYSEWWF5Lbb9aPMlA1EUse1HGwrLazu03ZEjRyIpKanjATZDiYEQQsJIFEXY3TyKT9X7ksO2H20oPlUPu5uX5cxBbjTGQAghYcRxHKZenQIAKD5Vj+JT9QCA8X0SMPXqFHAcF8nwAqLEQCLO/z4yenMM3H6zkgiJBt7k4E0KAFSbFABKDEQF/KekJlmtqKqqimA0hMjPe/nI37YfbapNDpQYSJvC0auWkM7Ef0zBe/nI+xjo2JlDfn4+vvnmG9hsNuTm5uKJJ57Avffe2+GYKTGQNoWjVy0hnQnHcbAY9JIxBe+Yg8Wg79AZw/r16+UKU4ISAyGEhNmk7CSIouhLAt7koMbLSABNVyWEEEU0TwJqTQoAJQZCCCHNUGIghBAiQWMMpE3h6lVLCFEvRRODIAhYsmQJUlJSsGTJEskyl8uFdevW4cSJE0hISMDChQuRlpamZHgkAJqSSkjno2hi2LlzJzIyMmC321ss++KLLxAXF4dXXnkFu3fvxubNm/GHP/xByfBIFKN6DBJtzp49i9///veoqqoCx3GYNWsW5s2bJ8u2FRtjqK6uxv79+zFx4sSAy0tKSnDjjTcC8NwxsLS0VJU3lyLa5KvHaPZfoGRBiBYYDAYsXboUu3btwvbt27Fp0yaUlZXJs21ZtsJg06ZNuO+++wKeLQCAzWZDamoqAECv1yM2Nhb19fXo0qWLUiESQkjY2Bt5nCxzoskhIMasQ98cEyyx+nZv76qrrsJVV10FAIiPj0d2djbOnz+PnJycDseqSGLYt28fEhMTkZmZiUOHDnVoW0VFRSgqKgIArFixAlarVY4Q28VgMER0/6HQSqzhirPWHOPpDteM3hyDpHbsr7Mfz3DQSqzeOC9cuABDgN+pQOyNPA7stcPt8hS51de5UVvD44YxCR1KDl6nT5/GoUOHcMMNN0hiMhgMiImJCfm4KpIYjh49ipKSEhw4cABOpxN2ux1r167FggULfOukpKSguroaqamp4HkejY2NSEhIaLGtvLw85OXl+R5H8oZrVg3d8E0rsYYrTt7R5GsZ6s/taGrX/jr78QwHrcTqjbOpqQl6PduHevlhO1xOARzH+S6Ru5wCyg83YuBQS4fiaWhowIMPPohnnnkGFosF7l9+zw0GA9xuN5qaWv6Od+/evc1tKpIYZs6ciZkzPQN8hw4dwvbt2yVJAQByc3Oxa9cu5OTkYM+ePRg0aJCqKwMJIYRVk0MIWPnc5BA6tF2Xy4Xf/e53uOuuuzB58uQObctfRAvctmzZgpKSEgDAhAkTcPnyZcyfPx87duzArFmzIhkaiTJcWjrQLaPFf1SPQZQQY9YF7PkcY27/R7AoivjjH/+Ifv364eGHH+5oiBKcqPGpPxUVFRHbt1ZOfQHtxEpxyksrcQLaidUbZ2NjI2JjY5leY2/kcWBPI1y/jDGIogijkcN1I2PbPcbw73//G3fddRcGDBjgOxtZsmSJb+an91JSoDhVcSmJdF78yieB6ostF6R2hX7RcuUDIiQCLLF6XDcyVtZZScOHD8fZsy1viS8HSgwkvKovAvWXIh0FIRFnidV3eKBZKXQTPUIIIRKUGAghpB20MjzbnjgpMRBCSDvodDpfzYBaud1u6HShf8zTGAMhhLSD2WyGw+FAU1OTKmuuYmJi4HK5YDabQ34tJQYSXqldQ3ueEI3gOA4Wi3oHkzsy/ZcSAwkrmpJKiPbQGAMhhBAJOmPoxPjFc4HLdS0XxHeB/oWNygfUAdSIh2iBHLfelvv23YFQYujMLtcBLmfg5zXG14in+fMRiIWQQJrfFqPuEo9amzuk22LIsQ0WdCmJEEIUcLLM6ftABzyD1y6XiJNlAb6chXEbLCgxEEKIAuS49Xa4bt/dXNBLSSdOnMD+/fvx008/+e7S17t3b1x33XXIysqSNRhCCIlWMWYd6i7xkg/2UG+9Lcc2WLSaGL777ju8//77sNvtGDhwIPr37w+LxQK73Y6zZ89i7dq1sFgsmDFjBoYOHSprUIQQEm365phQa3O3uPV23xyTottg0Wpi+PzzzzFv3jz069ev1RcfO3YM27Zto8SgVfFdWp2VpDVcWnrAgWZqxEPUQo5bb4fj9t2BUKOeDtBKYxFAO7FSnPLSSpyAdmKNhjhlbdTT2NgIh8MheS4lJSWUTRAVkWPuf7BtUH0BIdrDlBgOHjyI1157DRcvtuzEtWXLFtmDIsqQY+5/sG1QfQEh2sOUGDZs2ID/+I//wJgxY2AyyTvIQQghRF2YEoPL5cJNN93Urvt6E0II0RamT/rbbrsN27Zt00zHIkIIIe3HdMYwYsQIPP/889i6dSsSEhIky9atWxeWwAghhEQGU2J46aWXcPXVV2PUqFE0xhBF5Jj7H2wbVF9AiPYwJYbKykq88MILNMYQZeSYLhpsGzQllRDtYUoMw4YNQ2lpKQYPHhzueKKCEnP3Wfbhv06tOQa8o0n2OAhRAyV6FHQmzLOSXnzxRQwYMACJiYmSZY899lhYAtMyJebus+zDfx3eYADcbtnjICTSlOpR0JkwJYaePXuiZ8+e4Y6FEEJC1laPgoFDLRGOTpuYEsO0adPCHQchhLSLUj0KOhOm0eStW7fi2LFjkue8d1YlhJBIijHrWtRYhaNHQWfCdOR27tyJHj16SJ7r0aMHdu7cGZagCCGEVd8cE4xGzpccwtWjoDNhupTkdrthMEhXNRgMcDrl7TMaLZSYu8+yD/919OYYuP1mJRESLZTqUdCZMCWGzMxMFBYW4rbbbvM99+mnnyIzMzNsgWmZElNBWfbhv06SRu4hT0h7WGL1NNAsI6bEMGfOHDz33HMoLi7GVVddhQsXLqC2thZ//vOfmXbidDqxdOlSuN1u8DyPkSNHYvr06ZJ1du3ahXfffdfX3+HWW2/FxIkTQ3w7xB+/8kmg2nOr9It6PQSe9yxI7Qr9ouWS5RK/LAeUr8mgegtCIo95uurLL7+Mffv2obq6GiNGjEBubi7MZjPTToxGI5YuXQqz2Qy3242nn34aQ4cORU5OjmS90aNHY+7cuaG/CxJY9UWg/hIAQOA4oPlNEP2Wt0bpmgyqtyAk8pg7uJnNZowZM6ZdO+E4zpdEeJ4Hz/MtppcRQghRh1YTw6pVq3DnnXeiX79+rb742LFj2Lp1K5544omgOxIEAYsXL8b58+cxadIkZGdnt1hn7969OHLkCNLT0zFnzhxYrdYW6xQVFaGoqAgAsGLFioDrKMVgMER0/8Fc1Os9ZwoAOADiL//W6fWwWq2S5f68y4FfLu0YWv6a6M0xSJLpvfvvg+Pgm+gg5z7kpvafvZdW4gS0E2tniLPVxJCXl4eNGzeisbERAwcORPfu3WGxWGC323Hu3DkcOnQIcXFxmDFjBtOOdDodVq5ciYaGBqxatQqnT59Gr169fMtzc3MxZswYGI1GfPbZZygoKMDSpUsDxpWXl+d7HMkBVbU3BRd43nf5SPS7lCTwPKqqqiTLm7/O+754R5Pv0o4/t6NJtvfuvw+DwQD3L/+Wcx9yU/vP3ksrcQLaiTUa4uzevXubr201MQwdOhRDhw7F8ePHceDAAZSXl6OxsRFxcXHo3bs3Fi5ciL59+4YcbFxcHAYNGoTvvvtOkhj8+zxMnDgR7733XsjbJoQQ0nFBxxiysrKQlZXVoZ3U1dVBr9cjLi4OTqcTBw8exNSpUyXr1NTUIDk5GQBQUlLSoqCOtENqV98/dc1mJTVf3trrlK7JoHoLQiKPefC5I2pqalBQUABBECCKIkaNGoXc3Fxs2bIFWVlZGDZsGD755BOUlJRAr9cjPj4e+fn5SoQW1bxTToHAp5X+y1ujdE0G1VsQEnmcqPFGzhUVFRHbt1auNQLaiVXtcYqi5y6e3ji9j9VK7cfTn1ZijYY42z3GQNpPjqIwluKzULYRqMAt1GY/7XkvcrwPtSgsr4XdzWPq1Z4iTFEUse1HGywGPSZlJzFvh6WpDDWeIZFEiSEMZCkKYyg+C2UbgQrcQm3209o6rDFomSiKsLt5FJ+qBwDMtVqx7Ucbik/VY3yfBOYzB5amMtR4hkQac2KoqKjAqVOn4HA4JM9PmDBB9qAIURuO43xnCsWn6vGvM9/D7XZjfJ8ETL06hflyEktTGWo8QyKNKTH8/e9/x0cffYTevXsjJiZGsowSA+ksvMnBe9YAIKSkALA1laHGMyTSmBLDzp07sWzZMvTu3Tvc8RCiWt4xBX/bfrSFlBxizDrUXZLeEqZ5UxmWdQgJJ6bfNJPJhIyMjHDHQohqeZOCd0xh44whGN8nAcWn6rHtR1uLDmKtYWkqQ41nSKS1esYgCFdOW++55x68+eabmDZtGhITEyXr6XT0LaY5WYrCGIrPQtlGoAK3UJv9tLYOawxMz6sUx3GwGPSSMQXvmIPFoGc+Y2BpKkONZ0iktVrHcM899zBtYMuWLbIGFCqqY2CjlVjVHqdSdQzNt9ve/aj9ePrTSqzREGe76xjWrVvXsahIxAVrgKNEE55oE2hQWG6F5bWwN/LIFi1wOkSYzBzKOTsssaHVS3RUTbUTP+xrgqtJgDFGh2tzY5CcSpezOoNWE0PXrldO9T/++GPccccdLdbZsWMHbr/99vBERjosWAMcJZrwkNCIogh7I4+6cgHHjU1IshhQe8GNOpcAZLf/zCFUNdVO7PmqETwP6DjA7uCx56tGjPwVKDl0AkwDBB999FFIzxNC2ofjOGSLFiQZDah38vj5UhPqnTySjAZkixbFbr/xw74mX1IAPP/nec/zJPq1OV21tLQUgGcg2vtvrwsXLsBioWIbQuTmdIhIsngSg1eSxQCnQ7lzOVeT4EsKXjoOcDmplqIzaDMx/OUvfwEAOJ1O378Bz7eapKQkPPjgg+GNjpBOyGTmUHtB2hyp1u5GalpMK6+QnzFGB7uDlyQHQQSMJpqF2Bm0mRgKCgoAeAaiH3vsMUUCIqQzE0UR5ZwddS4BSSaDZ4zB7katy41yTsBAhS4nXZsbIxljEERAr/c8T6IfU+UzJQVtCtYAR4kmPCQ0HMd56hWygSwxBk6HiNS0GJRzAiyx7PUSHZWcasLIX3nGFFxOAUYTzUrqTFpNDI888gjTBvwvMRF1CdYAh6aktiRX/UBHTMpOarHf5mcKSsSZnGrC+FvCnwjUcMyJVKuJYf78+b5/Hzt2DF999RV+/etfo2vXrrh48SIKCwsxfvx4RYJUkhxz+4NtQ6keBcHqGIiUf78FjuPa3W9BDm3VS7DUOXj7OUC8AHDOsFVOB+sbEWy5kj0uCLtWE8PAgQN9/964cSP+9Kc/ISUlxffcddddh2XLlmHKlCnhjVBhcsztD7oNhXoUBKtjIFc077cw9eqUdvVbCDeWOgeHXfD1czAaAZfLFZZ+DsH6RgRbrmSPCxIapjEGm80Gs9ksec5sNsNms7XyCkK0pXm/Be+HVaj9FsLNW+dw3Oipb/BOaU0yGZAlxoDjOMX6OQTbT7DlSva4IKFhmns2bNgwvPDCCzh48CDOnDmD77//HqtWrUJubm644yNEMf4fVF5qSgpe3joHf/51Dkr1cwi2H5Y45Djm1L9CfkyJ4Xe/+x1ycnLw+uuvY/HixXj99dfRr18/PPTQQ+GOjxDFtNZvgfWW2koxmTnU2lvWOZjMng/HGLOuRczh6OcQbD8scchxzJV6v50J06Ukk8mEWbNmYdasWeGOh5CIaN5vwX+MAVDPmQNLnUPfHBNqbW64XOHt5+C/H+9gvf9+gi1vfsznjs3Bxq/LQj7mwfZDQtdqYjh8+LBvALr57TD8XXPNNfJHFUFyzO0Pug2FehQEq2MgV8jVbyHcWOoc/Ps5QDQBHMIySydY34hgy5XscUFC02o/hj/+8Y9YvXo1AODRRx8N/GKOi/jtuakfAxutxBrpOFnn1HeGOAVBkDTiav5YrjiV6nEhl0j/7FmFpR+DNykAV26NQdhEUy1EZ6NEvwU5OOxCWL8hv7S7Ak67gCnWFF+txPYqG0wWHR4f0/aHildheS3sl3ik2IxwNYkwxnCwpbhgSZTWKGjlmHcmTGMMJSUlGDBgAOLi4sIdT1SIploIoj7hnrcvCAKcdgGpVUaU1tqR3sWIc2dcSHUbUW11MZ05iKII+yUewnEOteBh0HNoaBQg1HCwZ/GqPyvo7JgSw/bt27FmzRqkp6dj4MCBGDhwIAYMGIAuXbqEOz5CSDPhnrev0+kwxZqC0lo77G4BJ2yesakuBgNGWxOYLidxHIcUmxG14CEAcPKerzQGcEiyGSgpqBxTYnj22WfhdDpRXl6Ow4cPo7CwEOvWrUNaWprkkhMhJPyUmLfvdIhI72L0JQUASO9iDKknhKtJhEHP+ZICABj0HFxOdU3/JS0xjyQJggC32w2XywWXy4W4uDhkZGSEMzZCSABKzNs3mTmcq3NJnjtX5/LVSrAwxnBw89I43bwIo4nOFtSO6YzhySefRG1tLfr374+BAwfi4YcfRo8ePcIdGyEkgHDP2xcEAdurbEh1G9HFYPCMMdS5UOd2Y3uVDQMEM9MYgy3FBaGGgwEcDHpPknDD8zyNMagbU2KIjY1FVVUVGhoafP/xPA+9nuYJBxJNtRBEfcI9b1+n08Fk0aHa6sJoa8IvtRIG36wk1jEGS6Ie9iweSTYDXE4R8Sadb1YSJQV1a7WOoTme53HixAkcOXIER44cwbFjx9CrVy/8+c9/DneMbaI6BjZaiZXilJcW6hjkiFVJ0RBnu+sYmrPb7aipqUF1dTWqqqrQ2NgIp9PJ9Fqn04mlS5fC7XaD53mMHDkS06dPl6zjcrmwbt06nDhxAgkJCVi4cCHS0tJYwyMqJkcjFpYPKTn2o5ZtKCFYnHLUF1CNgjYxJYYnnngC58+fR1ZWFgYMGIDZs2ejf//+iIlh6/9qNBqxdOlSmM1muN1uPP300xg6dChycnJ863zxxReIi4vDK6+8gt27d2Pz5s34wx/+0L531QqWwjM5itPkEKyALdT3EqlGPXI0v3lpdwUaXTyeGp8BnU4HQRCwrPgsYo16X7GVHPuRowEOyzaUcP6sA99/2wSBvwSdHhhyQwy6ZVy5dX6w48X6PpRokMOyDzU06tFKnCyYEsMDDzyA7OxsmEztG9ziOM7Xz4HnefA83+KbQ0lJCaZNmwYAGDlyJN58803Zv2mxFJ7JUZwmiyAFbKG+l0g06pGj+Y0gCGh08Th4vhHLis/iqfEZWFZ8FgfPN2Jwt1gIgmfqZkf3I0cDHJZtKPGN+fxZB77d7bjyg3YD3+524IYxQLcMc9CfiyAITO9DiQY5LPtQQ6MercTJiikxDBo0qMM7EgQBixcvxvnz5zFp0iRkZ2dLlttsNqSmpgIA9Ho9YmNjUV9fT0V0GiZH8xudTidJBjM+KAcADO4W6zuDANDh/cjRAIdlG0r4/tumltlf9DzfLcPM9HNheR9KNMhh2YcaGvVoJU5WzGMMHaXT6bBy5Uo0NDRg1apVOH36NHr16hXydoqKilBUVAQAWLFiBaxWK/Nra80xnm/OzejNMUj6ZTss63gZDIaQ9h+Ki3o9hAAfJDq9HlarNeT3wnGeeJuvo4S5Viv+deb7K4/H5rT6IdnaMX3pN1bk/WWP3+MhLcYYQtlPIJx4AdYEDpdtjb7nrAlmcKLJE5N4AUbjL+tynO94wrucZRsKEPjAZ5oCD0kMbR0vpvfhdzwk2vleA/7sWfYhcxzBaDpO1tfKHEtQcXFxGDRoEL777jtJYkhJSUF1dTVSU1PB8zwaGxuRkJDQ4vV5eXnIy8vzPQ5ldgDvaPJdTvHndjT5tsOyjlc4ZycIPA8EmDAm8DyqqqpCfi8GgwHuX/4d6L2Ei/fatdsv1o1fl7X6TT7QMfWOKfgfj8f//r3kjCHU/QSMlWtCVX2TpHisqt6BxBTRExPnhMvl8iUFt9vtWZe78nsYdBsK0OkBtPzVgE7vF2eQ48X0PvyOh1fz4xGKgH9PLPuQOY6ojvMXwWYlKdLiqK6uDg0NDQA8M5QOHjzYomo6NzcXu3btAgDs2bMHgwYNohkMGte8EcvqW3tjfJ8EFJ+qZ+7S5U0K3jGF96dnY3C3WN+YgyAIsuzH2wCn1uVGgkmPnokxSDDpf2mAY4coiuibY4LRyPm2F6jxTLBtKGHIDTFA8z8d7pfnEfznIggC0/sIdjzkwLIPJeKIljhZtXrGcOHCBaYNXHXVVUHXqampQUFBge+PeNSoUcjNzcWWLVuQlZWFYcOGYcKECVi3bh3mz5+P+Ph4LFy4kPlNsGIpPJOjOE0WQQrYQn0vkWjUI0cjFp1Oh1ijXjKm4B1ziDXqfWcMHd2PHA1wWLahhG4ZZtwwBr/MSkKLWUnBfi46nY7pfSjRIIdlH2po1KOVOFm1WuB2zz33MG1gy5YtsgYUKipwYxPJWOUocmpeaR+o8l6JgiyWpjLB4lCyVqKtn7sScYZCK39P0RBnuwvcIv2BHylqqWOIJh0tcgo05357Wa2kRiHQOh8frQmpjiFYrP77AK5ckgklDtZ6i7bmu8tWsxHkeEVTcZpW6gfUQpExBi3xzf1v9l+gZEHCz3/OvXe8wHt93O72NHxhWUcNcQiCwBSnd777xQsu1NcJuHjBhQN7GmFv5GV5r0ocLzVp63iSwJhmJfE8j8LCQhw+fBj19fWSZc8++2xYAiMEYK+F6GgdQ6hx/OvM93C73SHHwRJnsPnuctRshPt4qYmW6gfUgumM4e2330ZRUREGDhyIEydOYMSIEbh06ZIshW+EBOP/QebV/AOMZR01xMGyjWCNeOR4r0ocL7VQorFRtGFKDHv37sVTTz2FyZMnQ6/XY/LkyVi0aBEOHToU7vgI8V3q8Nd8GirLOmqIg2UbwRrxyPFelTheaqFEY6Now3Qpyel0+m5XYTKZ0NTUhIyMDJw6dSqcsRHSYs69/319gCuXboKt09Fvws3jmDs2Bxu/Lgspjjv6J+PjozVB42yrEQ/L8WC5L1S4j5eahLuxUTRiSgwZGRk4fvw4+vXrh8zMTHz44YewWCxISUkJ/mKNUU0dAwHAXgvR0ToGJeLQ6XRs2wgy312Wmo0wHy810VL9gFowNeo5duwYdDodMjMzce7cObzxxhuw2+2YPXs2BgwYoEScraI6BjZaibW1OFnm1LPUOnQUSx1DsDiUqGNgiVPJegoWWv8dVZuwN+rp16+f79/p6ekR79pGOh+HXWjzG1+gng3L/69C0rNBjrnsweb2s8QR7L2wxFprc+GHfU1wNQkwxuhwbW4MklM9l0ZY6i1Y3kswctRTqIlaekuoAfNN9EpLS/H111+jpqYGycnJGDNmDK699tpwxkYIgOD3sWfp2dDkEMN+L3y54gj2fmuqndjzVSN4HtBxgN3BY89XjRj5KyApxSjptTDXag25NwULOXptqIlaekuoBdOw/Pbt27FmzRrEx8fj+uuvR0JCAtauXYvt27eHOz5C2pyHDlzp2eC9ud6MD8p9H8beb+7BtiEHueIIts4P+5p8SQHw/J/nPc97xwu8N8Wb+/73kkFmuT6sm+/nj//8KSz7UYoSvx9K7EMuTIlhx44dePrpp3Hfffdh0qRJmDVrFp5++mns2LEj3PERwjQP3fuh7M//ttxKzWWXI45g67iaBF9S8O2XA1xO+eocWERTLYQSvx9aqqdgnsjbrVs3yWOWu6oSIgeWeei+ng1+vLflZt2GHOSII9g6xhgdhGZTRgQRMJrkq3NgEU21EEr8fmipnoIpomnTpmHDhg04d+4cnE4nKioq8Oqrr2L69OkQBMH3HyHhEOw+9iw9G5S4F75ccQRb59rcGOj18CUHQQT0es/zzWsUNs4YEnJvChbBejpoLTmopbeEWjANPr/++usAgN27d0ue//rrr/Haa6/5HnfWO7KS8Ao2D52lZ4MlFmGfyy5XHMHeb3KqCSN/5RlTcDkFGE3SWUlK1ChEWy2EWnpLqAVTHcPFixeZNta1ayvNZcKI6hjYdCRWJe/L35E6hmD9BeTs19CRfgxKYIlTzv209jgUWvl7ioY4Zalj8H7gC4KAS5cuITk5OcQQiVapYa46awzNP3z9H7+0uwLCZRHDuQS4mkQYYzj8W6yHLp7z1ReEEgfQen1AW3GwCjbfPdjyjtYosFJqP0RZTL+xDQ0NePnllzFr1iwsWLAAAFBSUoL3338/rMGRyFLDffvliEEQBAiXRfSpMcNWw6OpSYSthkefGjOEyyLT+JiSxyJY/wDqL0DCjXmMIS4uDuvXr8fjjz8OAMjJycE777yDGTNmhDVAEjlquG+/HDHodDoM5xJg4zwf4E1uz4e4gfM8z/KNvnkcrfVjkEOw/gHUX4CEG9MZww8//IAHHnhAcgmpS5cuuHTpUtgCI+qghrnqcsTgahJh0kvXN+k5uJzs3/SVOhbB5rtraT480SamxBAbG9uic1tVVRWNNXQCapirLkcMxhgOTl66vpMXYTSxf6grdSyCzXfX0nx4ok1Mv0kTJ07E6tWrUVpaClEUUVZWhoKCAtx8883hjo9EkBrmqssRgyAI+LdYD7fo+aYdY9CB4zi4Rc/zrGMMStQHAMHnu2tpPjzRJqYxhqlTp8JkMmHjxo3geR5/+ctfkJeXh8mTJ4c7PhJBapirLkcMOp0OungOp+DwzEpyiog36fBvsRG6eI55jEGpYxG0H4OG5sMTbWKqY1AzqmNgo/Y6hmDz7lliCLaOHP0alKoPkEtn+R1VUjTEKUsdQ2lpKdLS0pCWloba2lq899570Ol0mDlzJpKSkkIOmGhLuOeqs9QHBIshWK1DoOXby2pDrsdgORZauec+UZ5WfjeYxhg2btzoO91+++23wfM8OI7Dq6++GtbgSPSToz4g2DYEQVBNDQLpvLT0u8F0xmCz2WC1WsHzPL7//nusX78eBoMBDz/8cLjjI1FOjvoAlloHpeoxqMaAtEZLvxtMZwwWiwW1tbU4fPgwevToAbPZDABwu91hDY50DnLUBwTbhlpqEEjnpaXfDabEcOutt+LJJ5/E2rVrMWnSJADAjz/+iIyMjCCvJCQ4OeoDgm1DLTUIpPPS0u8G06WkO++8E8OHD4dOp/M17ElJScF//ud/hjU4Ev2a1wfMHZuDjV+XSXoJB/tW33wb/v2HAeCO/sn4+GhNq8vlPHPom2NCrc3tu2RANQbES0u/G0yJAWg5vSnYdCdCWMhRHxBsGzqdTjU1CKTz0tLvBnNiICRcJmUnSeoBvB/coXxgB9uGHPtgZYnVq24wkaiDVn43FEkMVVVVKCgoQG1tLTiOC1g1fejQIbz44otIS0sDAIwYMQJ33323EuF1ako24ekIljg7+lhNtPJzIdFJkcSg1+sxe/ZsZGZmwm63Y8mSJRg8eDB69OghWW/AgAFYsmSJEiFFhY4Wy6ihCU/zOICWBW6scWqleCiYwvJa2Bt5ZIsWOB0iTGYO5Zwdllhlfy7RcjxJ6BQZDk9OTkZmZiYAz9TXjIwM2Gy2IK8ibelosYwamvCwxMFanKal4qG2iKIIeyOPunIBx39qQn2d5/915QLsjcr9XKLleJL2UXyMobKyEidPnkS/fv1aLCsrK8OiRYuQnJyM2bNno2fPnkqHpxkdLZZRQxOeQHEEKnBjiVNLxUNt4TgO2aIFx41NqHfyqHd6PoiTTAZkiTGK/Vyi5XiS9lE0MTgcDqxevRr3338/YmNjJcv69u2L9evXw2w2Y//+/Vi5ciXWrl3bYhtFRUUoKioCAKxYsQJWq1WR2AMxGAyR2794AUZjoOdNAWNqLda5Viv+deb7K4/H5kTkWrY3Do7zxNo8jqBxhng8OiqcP3tOvABrAofLtkbfc9YEM7h2vJd2x6nw8QQi/PcUgs4Qp2KJwe12Y/Xq1Rg3bhxGjBjRYrl/orj++uuxceNG1NXVoUuXLpL18vLykJeX53scybscRvQui5wTLperxQAluMDHJFCs3ssy/hXsG78uU7xDm38cBoMBbrdbEgdTnCEej44K589e5JpQVd8kuWxUVe9AYooY8j7bHafCxxOIjruWqklH7q6qyBiDKIrYsGEDMjIycPvttwdcp7a21veHcOzYMQiCgISEBCXC06SONmtRQxOeQHE0b4AjCAJTnNHSvEYURZRzdtS63Egw6dEzMQYJJj1qXW6Uc3bFfi7RcjxJ+yhyxnD06FEUFxejV69eWLRoEQDg3nvv9WWzW265BXv27MGnn34KvV4Pk8mEhQsX0vS8NnS0WEYNTXhY4mAtTtNS8VBbOI7zxJwNZIkxcDpEpKbFoJwTYIlV7ucSLceTtA816umASJ9ShjLXvbVY1TJf3ts0xxtn8yY6aonTK9w/e7neb6R/R0OhlVijIU5VXEoi8issrw14k7jC8tqQtqOGoq+Xdldg+f9V+HovC4KA5f9XgZd2X0n6aohTSZ3t/RJ1ocSgQWqpQZCDIAhodPE4eL4Ry4rPQhAELCs+i4PnG9Ho4n3JghCiHLpXkgappQZBDjqdDk+Nz/Alg7y/7AFEEYO7xeKp8Rm+zoGEEOXQX51GKdV4Rgne5OCPkgIhkUN/eRqlVOMZJXgvH/nzXlYihCiPEoMGqaUGQQ7+YwqDu8Wi6JGRGNwtVjLmQAhRFiUGDWpt7v/4PgmK1iDIQafTIdaol4wpPDU+A4O7xSLWqKfLSYREAA0+axRr4xm1zf8P5PEx3cHzvC8J6HQ6PDmuu6SOobPRws+NRC9KDCESPv4rxMpzAIBacwx4RxMAgEtLh+6OmYrGEmyue7A+B2oRKM7tZbWqi1MpaumTQTovOk8PkVh5Djh/Fjh/FvyZ075/e5OFWmil1kErcSqFjgdRAzpjiFIsfQ7UQCtxKiWaalSIdtEZQxTTSq2DVuJUCh0PEmmUGKKYVmodtBKnUuh4kEijxBClgvU5UMuHjFbiVEo01agQ7aIxhhBxaenw/mnqzTFw+81KUhO19FsIRitxKoWOB1EDSgwh8p+SmqTy+7Kz1jpEmlbiBOSpLwi2DS0dDxKdKDFEOa3c118LcRaW18LeyCNbtIATL0DkmlDO2WGJZa8vYK1R0MLxINGLxhgIYSCKIuyNPOrKBRz/qQm1NU4c/6kJdeUC7I1s9QVUo0C0gs4YCGHAcRyyRQuOG5tQ7+Rx2dYIURSRZDIgS4xh+kZPNQpEK+iMgRBGToeIJIv0u1SSxQCng/2bPtUoEC2gxEAII5OZQ63dLXmu1u6Gycz+oU41CkQL6FISIQxEUUQ5Z0edS0CSyQBrghlV9Q7Uutwo5wQMFC1Bv/U3r1GYenWK7zFAZw5EPSgxEMKA4zhYYvVANjxjCqIJiSkiyjkBlli2+gKqUSBaQYmBEEb+9QXWX2pYWM4UWtsGQDUKRJ1ojIGQEMhRX0A1CkTtKDEQQgiRoMRACCFEghIDIYQQCUoMhBBCJCgxEEIIkaDEQAghRIISAyGEEAlFCtyqqqpQUFCA2tpacByHvLw8TJ48WbKOKIp46623cODAAcTExCA/Px+ZmZlKhBfV5GgsQwjpXBRJDHq9HrNnz0ZmZibsdjuWLFmCwYMHo0ePHr51Dhw4gPPnz2Pt2rUoLy/HG2+8gWXLlikRXtTybwoDoNWmMIQQ4k+RS0nJycm+b/8WiwUZGRmw2aR3mCwpKcH48ePBcRxycnLQ0NCAmpoaJcKLStQUhhDSXorfK6myshInT55Ev379JM/bbDZYrVbf49TUVNhsNiQnJysdYlRo3hTmX2e+h9vtpqYwhJCgFE0MDocDq1evxv3334/Y2Nh2baOoqAhFRUUAgBUrVkiSidIMBkNE989irtWKf535HhzniXfu2BxVJwUtHFOA4gwHrcTaGeJULDG43W6sXr0a48aNw4gRI1osT0lJQVVVle9xdXU1UlJSWqyXl5eHvLw832P/1yjNe4dNtfJePnK73TAYDHC73dj4dZmqzxjUfky9KE75aSXWaIize/fubb5WkTEGURSxYcMGZGRk4Pbbbw+4zrBhw1BcXAxRFFFWVobY2Fi6jNQBzZvCbJwxBOP7JEjGHAghJBBFzhiOHj2K4uJi9OrVC4sWLQIA3Hvvvb5sdsstt+C6667D/v37sWDBAphMJuTn5ysRWtSipjCEkPZSJDFcffXV+OCDD9pch+M4zJs3T4lwOg1qCkMIaQ+qfI5y1BSGEBIqSgyEEEIkKDEQQgiRoMRACCFEghIDIYQQCU6kCe2EEEL80BlDByxZsiTSITDTSqwUp7y0EiegnVg7Q5yUGAghhEhQYiCEECJBiaED/G/mp3ZaiZXilJdW4gS0E2tniJMGnwkhhEjQGQMhhBAJxTu4aZUgCFiyZAlSUlJajPbv2rUL7777rq9/xK233oqJEycqHuOjjz4Ks9kMnU4HvV6PFStWSJaLooi33noLBw4cQExMDPLz830tV9UW66FDh/Diiy8iLS0NADBixAjcfffdisfZ0NCADRs24OeffwbHcXjkkUeQk5PjW66WYxosTrUcz4qKCvz3f/+373FlZSWmT5+O2267zfecGo4pS5xqOaY7duzAF198AY7j0LNnT+Tn58NkMvmWu1wurFu3DidOnEBCQgIWLlzoi7lVImGyfft2cc2aNeLy5ctbLPvyyy/FN954IwJRSeXn54uXLl1qdfm+ffvE559/XhQEQTx69Kj45JNPKhidVLBYS0tLAx5rpb3yyitiUVGRKIqi6HK5xMuXL0uWq+WYBotTLcfTH8/z4rx588TKykrJ82o5pl6txamGY1pdXS3m5+eLTU1NoiiK4urVq8Uvv/xSss4///lP8dVXXxVFURS//vpr8aWXXgq6XbqUxKC6uhr79++PyFmAnEpKSjB+/HhwHIecnBw0NDSgpqYm0mGpVmNjI44cOYIJEyYA8LRKjIuLk6yjhmPKEqca/fDDD+jWrRu6du0qeV4Nx9Rfa3GqhSAIcDqd4HkeTqezRYOzkpIS3HjjjQCAkSNHorS0NGijLrqUxGDTpk247777YLfbW11n7969OHLkCNLT0zFnzpyI9YR9/vnnAQA333xzi1kJNptNEldqaipsNlvEOuW1FSsAlJWVYdGiRUhOTsbs2bPRs2dPReOrrKxEly5dsH79evz000/IzMzE/fffD7PZ7FtHDceUJU4g8sezud27d2PMmDEtnlfDMfXXWpxA5I9pSkoKpkyZgkceeQQmkwlDhgzBkCFDJOvYbDakpqYCAPR6PWJjY1FfX48uXbq0ul06Ywhi3759SExMbPMaZ25uLgoKCrBq1SoMHjwYBQUFCkZ4xX/913/hhRdewFNPPYXCwkIcPnw4InGwCBZr3759sX79eqxcuRK33norVq5cqXiMPM/j5MmTuOWWW/Diiy8iJiYGW7duVTyOYFjiVMPx9Od2u7Fv3z6MHDkyonEE01acajimly9fxrfffouCggK8+uqrcDgcKC4u7vB2KTEEcfToUZSUlODRRx/FmjVrUFpairVr10rWSUhIgNFoBABMnDgRJ06ciESovsHvxMRE3HDDDTh27FiL5f7Nwaurq32vUVqwWGNjY33feK+//nrwPI+6ujpFY0xNTUVqaiqys7MBeE7DT548KVlHDceUJU41HE9/Bw4cQN++fZGUlNRimRqOqVdbcarhmP7www9IS0tDly5dYDAYMGLECJSVlUnWSUlJQXV1NQDPl4jGxkYkJCS0uV1KDEHMnDkTGzZsQEFBARYuXIhrrrkGCxYskKzjf/2zpKQEPXr0UDpMOBwO36Uuh8OBgwcPolevXpJ1hg0bhuLiYoiiiLKyMsTGxkbk9Jwl1traWt910GPHjkEQhKC/zHJLSkpCamoqKioqAHj+CJv/bNVwTFniVMPx9NfW5Rk1HFOvtuJUwzG1Wq0oLy9HU1MTRFHEDz/8gIyMDMk6ubm52LVrFwBgz549GDRoUNBOjjTG0E5btmxBVlYWhg0bhk8++QQlJSXQ6/WIj49Hfn6+4vFcunQJq1atAuD5VjB27FgMHToUn376KQDglltuwXXXXYf9+/djwYIFMJlMEYmTNdY9e/bg008/hV6vh8lkwsKFCyPSlvTBBx/E2rVr4Xa7kZaWhvz8fFUe02BxquV4Ale+DDz00EO+59R4TIPFqYZjmp2djZEjR2Lx4sXQ6/Xo06cP8vLyJJ9PEyZMwLp16zB//nzEx8dj4cKFQbdLlc+EEEIk6FISIYQQCUoMhBBCJCgxEEIIkaDEQAghRIISAyGEEAlKDKRTmz59Os6fPx9w2TPPPIPPP/9c4Yg82oorkDNnzmDJkiVB74ETqnfeecc3RZN0HpQYCIkwORLQ+++/jylTpsg+j37KlCn4xz/+AbfbLet2ibpRYiBE42pqanDo0CHccMMNsm87OTkZ3bt3R0lJiezbJupFlc9ENbZu3YpPPvkEdrsdycnJmDdvHq699loIgoCPP/4Yn3/+ORoaGnDNNdfgoYceQnx8PCorK/HYY4/hoYcewocffghRFHH77bfjjjvuAOC5VcFbb72Fs2fPwmQyYcSIEZgzZw4MhtB/9b/44gts374dtbW16NevHx566CHfrZinT5+OefPmYceOHairq8PYsWMxd+5ccBwHQRDw3nvv4auvvoLZbMaUKVPw5ptv4m9/+xs++OADHDlyBOXl5di0aRNuvPFGzJ07FwBw8OBBLFu2rMX2mjt48CAyMzMlzVmqqqqwadMmHDlyBKIoYsyYMZg7dy527dqFzz//HFlZWdi1axfi4+Mxf/58nDt3Dlu2bIHL5cJ9993nu00zAAwaNAj79+9X/Q3viHwoMRBVqKioQGFhIZYvX46UlBRUVlZCEAQAwD//+U98++23eOaZZ9ClSxe89dZbeOONNySl/aWlpXj55ZdRWVmJZ599Fn369MHgwYOh0+kwZ84cZGVlobq6GsuXL0dhYaGkExeLb7/9Fv/4xz+wePFipKenY+vWrXj55Zfx3HPP+dbZv38/li9fDrvdjsWLF2PYsGEYOnQoioqKcODAAd+dT/07g9177704evQoxo0b16LfR2vba+706dNIT0/3PRYEAS+88AIGDRqEgoIC6HQ6yY0dy8vLMWHCBLz55pv44IMPsGbNGuTm5mLt2rU4fPgwVq9ejZEjR/puEJeRkYG9e/eGdLyIttGlJKIKOp0OLpcLZ86c8d3vp1u3bgCAzz77DDNmzEBqaiqMRiOmTZuGvXv3gud53+unTZsGs9mMXr164aabbsLu3bsBAJmZmcjJyYFer0daWhry8vLadTvyzz77DHfddRd69OgBvV6Pu+66C6dOncLFixd969x5552Ii4uD1WrFoEGDcOrUKQDAN998g8mTJyM1NRXx8fGYOnUq0z5b215zDQ0NsFgsvsfHjh2DzWbD7NmzYTabYTKZcPXVV/uWp6Wl4aabboJOp8Po0aNRXV2Nu+++G0ajEUOGDIHBYJAMfFssFjQ0NIRwtIjW0RkDUYVu3brh/vvvx4cffogzZ85gyJAh+O1vf4uUlBRcvHgRq1atklxG0el0uHTpku+xtxEJ4Lnj5OnTpwF4zkTeeecdHD9+3Nflqj39gy9evIi33noL77zzju85URRhs9l8l5P8b80cExMDh8MBwDMG0Dw+Fq1tr7n4+HhJE6mqqip07doVer0+4PqJiYm+f3svP/nvy2QySfZlt9s10RGOyIcSA1GNsWPHYuzYsWhsbMRrr72GzZs3Y/78+UhNTcUjjzwi+dbrVVlZCcBzz37v7Yarqqp8t2l+44030KdPH/z+97+HxWLB//7v/2LPnj0hx2a1WvGb3/wG48aNC/m1ycnJsNlsvsf+vQYAdHgmUa9evfDVV1/5HlutVlRVVYHn+VaTQyjOnj2L3r17d3g7RDvoUhJRhYqKCpSWlsLlcsFkMsFkMvk+MG+++Wa8//77vss2dXV1+PbbbyWv/+ijj9DU1ISff/4Zu3btwujRowF4vu16G6qcPXu23XPyb775ZmzduhU///wzAE+f5W+++YbptaNGjcLOnTths9nQ0NCAbdu2SZYnJibiwoUL7YoLAAYPHoyTJ0/C6XQCAPr164fk5GRs3rwZDocDTqcTP/74Y7u3f/jw4YBjGyR60RkDUQWXy4XNmzfj7Nmz0Ov16N+/v+8++JMnTwYAPPfcc6ipqUFiYiJGjRolmZ45cOBALFiwAIIgYMqUKb6+t7Nnz8Zrr72Gbdu2oW/fvhg9ejRKS0tDjm/48OFwOBxYs2YNqqqqEBsbi2uvvRajRo0K+tqJEyeioqICTzzxBCwWC37961/j8OHD0Ol0vvdXUFCAzz77DOPGjcODDz4YUmxJSUm45pprUFJSgtGjR0On02Hx4sV48803kZ+fD47jMGbMmIBnXMHU1NTgzJkzGD58eMivJdpF/RiIpnmnq/7tb3+T5bKJEg4cOIDXX38d69evl22bZ86cQUFBAZYtWyZrkds777yDq666CpMmTZJtm0T96IyBkDBzOp0oLS3FkCFDcOnSJfzP//yP7N/Ae/TogeXLl8u6TQD47W9/K/s2ifpRYiAkzERRxIcffog1a9bAZDLh+uuvx/Tp0yMdFiGtoktJhBBCJGhWEiGEEAlKDIQQQiQoMRBCCJGgxEAIIUSCEgMhhBAJSgyEEEIk/h+etkSio0GtdgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 可视化特征\n",
    "marker = ['s','x','o']\n",
    "for _,c in enumerate(np.unique(y)):\n",
    "    plt.scatter(x=iris_data.loc[y==c,\"sepal length (cm)\"],y=iris_data.loc[y==c,\"sepal width (cm)\"],alpha=0.8,label=c,marker=marker[c])\n",
    "plt.xlabel(\"sepal length (cm)\")\n",
    "plt.ylabel(\"sepal width (cm)\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们可以看到：每种不同的颜色和点的样式为一种类型的鸢尾花，数据集有三种不同类型的鸢尾花。因此因变量是一个类别变量，因此通过特征预测鸢尾花类别的问题是一个分类问题。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "各个特征的相关解释：\n",
    "   - sepal length (cm)：花萼长度(厘米)\n",
    "   - sepal width (cm)：花萼宽度(厘米)\n",
    "   - petal length (cm)：花瓣长度(厘米)\n",
    "   - petal width (cm)：花瓣宽度(厘米)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.3 无监督学习"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们可以使用sklearn生成符合自身需求的数据集，下面我们用其中几个函数例子来生成无因变量的数据集：   \n",
    "https://scikit-learn.org/stable/modules/classes.html?highlight=datasets#module-sklearn.datasets    \n",
    "![jupyter](./1.2.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-14T11:34:32.719213Z",
     "start_time": "2021-03-14T11:34:32.578215Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAD4CAYAAADhNOGaAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAACF90lEQVR4nO29eZxU1Z33/z631t67q5u9aXpBiLihIYnjvvDkyWSZOEkmyWSyR40CGhWMrGpEBBfAUSCJS8zEPDNPfpNJjCYTJ+ExYNQsGsUFEOkVaNbu2rq6a733/P44dW8tXc1iQy/0fb9evOiqulV16tat8z3nu3y+QkopsbGxsbEZs2jDPQAbGxsbm+HFNgQ2NjY2YxzbENjY2NiMcWxDYGNjYzPGsQ2BjY2NzRjHNgQ2NjY2YxzncA/g/bJ///5T/h41NTV0dXWd8vc5FdhjHx7ssQ8P9tiPj8mTJxe8394R2NjY2IxxbENgY2NjM8axDYGNjY3NGMc2BDY2NjZjHNsQ2NjY2IxxbENgY2NjM8axDYHNSUFKiYzHsMVsbWxGH6O2jsBm5CClxNi4CjpaYFoT2vxlCCGG7L1JxMHtGbL3tLE53bANgc3gScSVEQh2Z257vKf8bYfTANnYnE7YhsCmICe00nZ7YFqT+ntak7o9iPc8bvfSMBkgG5vTDdsQ2PTjRFfaQgi0+cuOajjMGAKA8Hj7HZP9nsEzzkRed/uQGSAbm7GObQhs+nOSV9rWJL/9DXV71my0BcsRQli7AMMwoKMZgn5SrbuO6z2PxwDZ2NgcG9sQ2PTnOFfalivH5UZuum/gHUQirib5VFLd7mhRz3N7lIFobwZDh2gfOF04G2eSOs7VvRDCMhh24NjG5v1hGwKbfhyvq8dyH02thz1tEPKrB9OTvPl83B6omw4RtSOwjIu58wj5QQiQEip8VNx6N/7evpz3KjSWHEMUj8FjD8DedqifbgeObWxOANsQ2BQke6UNBSbjfPfR1AY1mU9rUhPzhnvVSn9qA+LmO9EWLOsXI5BuD9Q1gmEoI+BwKKPi8SIjvUfdbeQYIl2HvgjoKTWWtt3InhCUVdjGwMbmOLANgc0xKRQ87uc+unEJorcHWVKGDAdVPEBPwfYA8tF7EDffheYtynlN0zCgCahtAiTsaSO4+g6MZFK9X2097GvvH6/INkTmbsJ6cQO58laknVJqY3Nc2IbA5tgUCB4Lj9dyH5k7ANneDEjo682szgH2tqtJXwhwe5QR2HAv7GmF3h4VO5CAAELpYLFhQNCvJvi6BvX/lGkYhoEwDKRhQO009frmjkAImHkWdO7tZziy3UgimbDjCDY2WdiG4DTmpAVP3R7l+jEM9X86kGu6d2TQD2+/Vvi5ThfUNSIffwhpxhNA7RjMVbzDqVxEQgNacNZPJ5nSAQF1TfCtW+HxB+Hdt+G2ryCLipWxEQJmngM3rVCTOyBdbvjuN9Rjhq5uZ+9oDB0pNDuOYGOThW0ITlNOpBbgeHL86WiGnhC0NyOlzPXTp9NCLZwuNcl7vKA5QBrQ3go9QQgH0vdluXJKyxDX3652C4/cQ7JtN9Q2wD0b4fGHYMU8iIRVZhGocZjsa0dLJTPuomRCTfRSgtAsA9HPjZTOXLIL0GxsTpIh2LRpE6+//joVFRWsXbu23+NSSp566ineeOMNPB4P8+bNo7GxEYAtW7bwi1/8AoDPfOYzXHHFFSdjSGOKgiv/eGzAWoDs4wGMDffCjm3qsVnnoy3IGA3DMJAH9qnJV0qIhJA9IWR5JUYoAG3vZdJCAUrKlStn1zvKXQMqo8gwzMFmJnSHQx1ffwbC41XGaNfb6vV6whAOwc43M6/vcOa6nJyuTHDaTEOdUpcTuLZSX814hqGrnUf6ecRjtrvIZsxzUgzBFVdcwcc+9jE2btxY8PE33niDgwcP8sgjj7B7926eeOIJ7rvvPiKRCD//+c9Zs2YNAIsXL2bOnDmUlpaejGGNCQZc+Xu8BWsB8o8X1y5Uf5uT7Z5my2gYhoG8/esQDma/ITyxDsPphHf+ljuYkjJY9QO4a0FmsgcVBzB3AEKoHYGeUkbgjvsRNePVDiP7tfQU3H9HrpEpKgHUSp+6erjuuwiHQxmQ7W+oY0N+KK2A+36I5i2yJnYxbylEwsiiEhV7qKpGbrpPxTWkgdQctl6RzZjlpBiCWbNmcfjw4QEff+2117jssssQQjBjxgx6e3sJBAJs376dc88915r4zz33XLZt28Yll1xyMoY1NhigCnjAWoD840H553eE039Pz6yiI+FcN4zJu2+m/fl5OJ1qYp7WBNvDmfv1FCoSDBSVgqZBJKTcRCtuRJ79QeSCZWpXMOt8tL0tGD1h5UrKJhKC8iqYVAt72mHFPKTDoW5nu5oiIRXQLipWxqwnBD/ZoD53JKzGU1ahdhjZNQxZ5y8bu1DN5nRnSGIEfr+fmpoa63Z1dTV+vx+/3091dbV1v8/nw+/3F3yNzZs3s3nzZgDWrFmT83qnCqfTOSTvMxiklATPOJNU6y6cjTOpnDwFIQROp5Nx48Yd8/iKyVMIeb2kKipx1p9BxbIH0TRN7RyKi+gur0SGQ1BalmsUhCBnCS8E7jNmUTmlFnnXekL33U6qbTdS15Gaplw9qSTE+lTMwMTQ0fa1UlNWivAWIe9eD4k43ffdgd6+O3c3Asp4hAO592UbNTUYnP/xQyoWr8F/7achlHc8QE8I17lzSGkCDAOhaerzT5qMpikjZxgGMhQguPF+jPb31Pldcv9RjcFouGYGwh778DASxj5qgsVz585l7ty51u2urq5T/p41NTVD8j6DRV53OyTipNweurvVpFho7FYK5bWLEMmEOv7AfozdOyHYTUI36NrfifB4c4u1SspgaiPsbUuv0gXMOh+uW6hiEQDeIlLeIrq7u5HxGEbLe2qCrqyGZevg3zbAzm25rp40xpQGunoiiEiv0hy645vIUAC8xSd+MkrKoTdMqvldupfPK2wEADQHyfYWqK0DKZC73iax7a8c+d5tqvhNSuUW6wmlDZ7MOT8DMVqumULYYx8ehnLskydPLnj/kBgCn8+X80G7u7vx+Xz4fD527Nhh3e/3+5k1a9ZQDOm0Ir8KuBBWbCBd7ctNK9DM6t6pjcpl0tuDfHwt8trbLAE4i11vw9qfqInc5UZomsowKi6xXt+SlsgrNtPKK5HXL0IuvzEjQ2FSUQVfW5C5HQkjQ0Hlqon2Zn/IXPdPIUoroOEMZbCmTFOB5oHQNLWz2JFnKNp2YcSiEOnJ3Y2UVkDtNCsd1XYV2ZxODIkhmDNnDs8//zwXX3wxu3fvpri4mKqqKmbPns1//Md/EImo7JI333yTL33pS0MxpLFHIg5tu9XkF/IjN6xE3nSnekzqym8uJexpUZo9kR6VlWOu4PWUuv/678IP70fu60DWT0dct0hNjhvuVRNwOj+/X3zC41WxgzZDvZZZYCYl3HMLRl0j4qYVUFqOKK9QOwITs17g23eo+oEHFvc3KOWVcO8PVPWyqWZ6y78MfD4K7EwAtQNYdoPKaHI6IZVSwe2+CLz7NnLDKqQm7GY4NqcVJ8UQPPzww+zYsYOenh5uuOEGPv/5z5NKqTS/j370o5x//vm8/vrr3HzzzbjdbubNmwdAaWkpn/3sZ1myZAkAn/vc5+yMoUGQnxaa85jLneub3/EmMh5Tk9je9nTevVDpl3vb1USpOXJfZMebcEuWoX7rVeSyG9TrRsLqNdqbrcpjc5ei6zp0H8ld5Z95HnztJlh5qzJO219HbliJdtOd+J58lq7l81UBGVI979234K758MBTKj317TxDMG16joQFbo9yafUEVWA7+7Nnk5+SCirYnBbAY/wkaN6RSXvtaFbnKaSqnk1NI+DEmurY2IwgToohuOWWW476uBCCa6+9tuBjV111FVddddXJGMZpyfFmrOSnhcq71uc8LpIJlSKZ/zyXW+n5GAZMa4R5y2DDPWrS643kH93/jcN5K/esymNIG4H5/9R/st3XpoLDdQ3wTkBNtDvfQsZjOBwOxPxl/V1JPSHwd8H85fDI95Trx5ywr1ukXDqPPQh7W9OVyiIzLqGpz5jtYvIWw92PwtObVPzCTHmVUsU29FTGCIAyGoYOsaj1t1x5K7LO1ElqPf6mOjY2I4gCOYA2IwVzcjeW34ixcZW12pRSYsSi6NE+jFg0kyJppoV2tGSCuOZrudyZoi5QK3K3R2n+7HpbrdSFBnd8S638e8LgLSqcJpqNw6kmV6cLzjofcdOK3Emw+0heEZgzLTsxXe0arv+uup2H8HihfrqakJ3p9YrDoWoLvr8abr4Lzr5APT5tuqpAvu0rqrYhFFAFculdKYae+ex3b8h8plgfLL9Bid6t3JQ7gHETld3LXuHrKWUcU8l0cV04fb6brXNvNtUxq7XtHYLNaGDUZA2NSQrUCEi3J1MJrKfUKr84nZsvjfTE2KTcMpFMsFUkE8jsCdpIIWNR63UANaFFzBRRqfzimiMtCCegtBwMCb1ZNQKgJsWSUsS377BSLy1Ky3Jvr3oMUVxiSVkIjxc542wlQDetMUfHyIwzGE6XMij336F2CB0tSlbipjvVZBuPwT235Pr9dR36evqf03//Qe7tVFrltKxCBYTNz9+8Q51XBDk7IWko4ydlrjtNV8bG0TCDlFnpbMcRbEYJtiEYyRTqFJaIKz98dhA3y6ctVqxHlFWoXUM4CKXlanI2tf/NauCdb8EPVue+39RG5YvPXsELoQKxtfVw052IVBL5gzXKgEipJkAhwJDqPQ0jR65BKyrGOGdOZlKsqs7oFMWiyMceVDsSXYdd7yA33qdqCciI2ol4DPl/H1OGyelSQnRmpfQTa9VrS0M9Zga9Tcoqc3sVHOyED5ytPr/1udPurBwpa6kqogth6FBcpoyvoVvyHBY5BlztDER2/MLGZoRhG4IRTKHqYGkqgW7P8s1rDrWinVqv+gH09XJk4ddUoLS8EuPBH6NpGuKmFchH7lauHylh1/aM5MMHzoHrFin3iklZpXLPIFUA+QdrEPOXwc13ITesVBpCpox0tBeWfhsphKr2ndaEmLcUkUwgFizPMQ6ZeEazyk4yJ+lUUt2Xdmv1Oy6VVMbuuoXqdbL1lCqrYc0Tyhgsv0GNq7RCSV643CrusW+P+jzfvFV9Tj2lVvff/q7qpVBoB1EIKZVbqbikn7FI7t6hWm7WTlOuo0iPUl5N92i2sRmJ2IZghJNfIyCEgJtWwMMr1EQupXLZTJ6qJuaFX8t13YSDyO4jSFPP59uL4Yf3qxVxth7Q3nZ44qHMKtfpgrv+VbluVszLcU9pHi/SdMs8/lDaH5/Mcisp5KMrkfvalVHIdo9YK2a/eh/NkRmLYShjR2//4yp8UD89U9CVvWOqa1LuJo8XufYnahI2d0OA/M73coX2zjo/s0vxFiE9XmVMe0Iq22jadJUOm1/FbKKnCstvhINwxzczCquGrtxettKpzQjGNgQjlKP26f3+ajiwXxkAh7N/z+B8Vt+O0XCGWv13tMDkOuX/zp64e4Kwr0Pl6+9rtybW3Mm2Ub1/WoZaeIswrlukUkizJ0ynq2AfYwaYwPnyjbDyFjWxahoiES94nLhuYY5MtrljMg2SXDEPae5EPN5M1hAFDOr8Zap4raRMpbu6Pcj7fwSP3K0a2zgcsHyd6m1QkLzYQeYLUv/rKRWUTrvVDMNAxKIFZb5tbIYb2xCMQI7aSyC74Xs6JiBLylRBl5SFV7A9QfUcUM8L+dWq18zWKSpWK9j0JEoibk2sZhtKImF4eqM12ZpjEh4vsn56ZlcAyshcdzs8uc76DNkppdkuL0tCOtqXlpWebgW6BxTOy0IIAUIg97Rau5bsnUihQK2UMtMH2dAxhKbe19BV7YLZr0DTcncrJppDPWZ+Xk0Db0lucNoMKIeD6tzc9hWVhDRrNuK6RbZBsBlR2IZgJDKAoijQL4AsS8pg46r0Kr5RTVrtLRCNqInIzCqqrVcr5HDAatrC6sdVLr/LjejtsVwpMm9i5ZHvqd1CXyQz+WWrnC5Yns7hf0C5mOrTxV1HmcStFXo8pt4rz//f77ijkX1O8nYiBZvYF+p3vKc5ky6arocQZRXIWbP7y20Xl6oYwK600SirVJ919aLMjuC7a2DN7epvQ88Ykx3bkMtvRNod0mxGELYhGIkUyhZK0y+tMruwyqzuNf+BmoCkoQxF7TSYcTY071TB3ac3wfylYOryT23AuGkFIvv99VQmK8b00xcYk6OoGHnzXf0n/hOZxKc1HVXQbSD67TCyVvvynlv6T7puj3JJSanqCzRNyW8jYU86HXTBcjRNw7hphQqMm9XWQlNeof17lWvO7MD26ErlTtINlTI7bqKS3DZ3CWa9BVgpsHbcwGakYBuCEUi273sgpJSwMdNZzFrZ5lfwQiazJeTPuINSSaUrFAmrdNSQH8IB5IaViJvuVO/fE0Lec0v2m6qV8rylA7tpjjKxFYp7HI/753gw31sAMnvsA066aUNZPx1x/e3KxSUl8tGVKkj8/dXI+cvUDummOy0jQySMXHmr2k1U+NTK/4E71E6rrBJnfROpd96A276cPTr13kJTxWsOZz9jamMznNiGYAQjH38IOlqUT37BcnVfPKZy59ubc1MXzR7Bfb1qhStEYaOQSqqJqDK9si8tV+mopstob3tGK6isQvn/zfdKJWFfu0oFPcGV7NHiHsfl/jkBhBCZsReIUZCIq0yekB/2ChXrEEJN9vva+xmPHCNTVqHE8wwDpkyFn25S1cbptplGZ0cBXSOZadtZVgnL16GVV6pH4rHM2GxFU5thwjYEIxQZj2UCsDt6lA/+yXVqUja7bAE52St9ETXJl5ThOuNMkp/5Gty/ODeA7HDCrNnWKlgIkev+yJo0c7Ny1qodxPtdyR4t7nEKOOpOYyDX2wD399vJ3LgEbv+6qscwqfDB9bfj+MmjGP6jaMtHe+EnG5Hzl2ZcWHVNynjsaVVG0q45sBlibEMwWsjOFiJrksjPYtRTEA6o7l5lFf0zXu7ZhDZuQs5Ek+3+yJ80zTRRuWCQ7pujxD1OFQPtNAYyEoVccoZhZNxFZqyhtwcZyZPZqGtA8xZRedv3OHLj59X3VFYJk6bAezvShXua5ZKT3UcyhlEamcK67WGMWNSqBrcNgs1QYBuCEYglVHbmbNjbogTayiqQdY0q1z7b5ZMvauZwQmk5joYZGIl4rpR0WYXVKD6fY7lnBuu+OVmxgJPF0T6PKVsh6xrVJL399RyJbUrLVdWy1a1ttlJETcSRbp96jkhb6AOdWJa6qDijYLrmu+r+ymrVRGf7G+oYPQU/fEC5mKbWIxas6K/fZGNzkrENwQgjx5de14RY+f2MC+e6RcjlNwzcflFoUFQCtfVqolt5K5CWaa6tg5vuGnBSGYoG7Sc7FnBKyHZhmUY2K6UUtwdNCIyHfqxSU82ObBtXYextI9w0AxCZFN3JtfBuMKNd9IGzoXOfcteVV8LSh9RrL7kuk5pr6j2F/Kom4uY7h91w2pze2IZgBCHNRifZvnSwVEeFx4ucNh3eeV2tKoWmctpjfZljIyHY107KoVlFZ9xxP6J63FGNwIAFbGONvEpqECo2kl6dm+dF0zSoqFLn7pF7rF1Dqr1ZPW9vm9oZHExXgJupve/thBmzlCEIB5UBKClTuwXzGMhkge1ts9NMbU45tiEYIeT0FDblpOuaVIWvGUScv0y1hjR3BdJQ6Yil5WoF2derXA+19TiKijCMtNbN/Xcg66cjB5rghziQO5LJd2EBR98pJeLpSV/tGpz100lef0f/NFOzQlnXVXGeSTqmQ4UPzjxX6UdBRt00T9bDxuZUYDsfhxmrgYmppBnyg9CUnPR1CzMTdHYLyKmNmRfoCannRMLpiSalUjyFUFo5QstNhyz03i63WgWbvQzGeH67KZ1hppUeVQ7C7VGKphU+OOsCKpY+qJReyyoy53RKnVrxAyBzxeocTktMj28vVsVoegriUZUI0N6MXDEvpzGRjc3Jxt4RDCP58QDlikBV2KZ7CliBR2koKQhQctHLb1CTv8OhVo/hIFZQMuQn1borowNUIJc+3x1kSkaPhEDuaCJ/B2G633LTTLdlqoqzeySXlMOdDyOKS1S3uE33ZdqDppLqO81urTmGd2o2p5aTYgi2bdvGU089hWEYXH311VxzzTU5j//4xz9m+3a15U0kEoRCIX784x8D8IUvfIG6ujoAampquOOOO07GkEYFahfQrGSWkSq1M502KKVEdh3O6N9oDiUrYRaTRfvSEtQVauX/bxugY7eaSAwdmUqpiWmgTJ08d5BIJt6XvIPNUYLgkXBm9Z+9mi8pU99fbw8s+zZy1vnwrVtzGw6ZmC4lQ88sBGxsTjKDNgSGYfDkk0+yfPlyqqurWbJkCXPmzKG2ttY65utf/7r1929/+1va2tqs2263mwcffHCwwxh1SClV5XCkJy0GF4JN92J85x5VtXr71zOTiOZQOwbIrSUQAgwdUVaBuGm56kh2x7dASmRvBJGIq85YhSapYcjrH0tIKeHpjbkreki7fWJQUqq+31QS2neroLFVfVwBCJVxtH+fSlMV2vuq6LaxOR4GbQiam5uZOHEiEyZMAOCiiy7i1VdfzTEE2bz88st8/vOfH+zbjlrMNE3DMNSkbq4ApQ7vvgMLPg9nnJnrRy4pg6/OV758q4ZApnsJa5nVvNuTKSDTU0f1KY+0vP7TDnPHJaWSnygqSdcdoO6rnaYKzXQ9V9UVYPl6ZUR2vqkeN+W5bWNtc4oYtCHw+/1UV1dbt6urq9m9e3fBY48cOcLhw4c5++yzrfuSySSLFy/G4XDw6U9/mg9/+MMFn7t582Y2b94MwJo1a6ipqRns0I+J0+kc9PtIKdUKML2SC65eTKJ5h1Kp7IuA0wmprAIxPaUMQu6rIO5bqDJSolmpok4nzsYZVE2arJQyo30ccTohmQSnk5rqahzFJYMa/3BwMs77cGGOXUpJ8IwzSbXuwtk4k7Lv3Il//heRpt//wD4cZ56LsX8PMk+SospXTaCz3TL6oqycmsX3oaUDztnX1Mk04KfDeR+NjISxD2mw+OWXX+bCCy/MyWfftGkTPp+PQ4cOcc8991BXV8fEiRP7PXfu3LnMnTvXut3VdRQ9l5NETU3NoN6nX0D22oXI97bndhIzc8xzn5n5c8bZcLAT6e8imdL7HZZqfY8j99ymVvcAHzgv3Vgeuu5fhrZg9NUEDPa8DyfZY5fX3Q6JOCm3h0AwiEwH/dElBP3oQb/6/iuqIBTE/N4D6+/OWhwIZG0j3ZFeRG+fkrzYcK9KWT3JNR+ny3kfbQzl2CdPnlzw/kGnj/p8Prq7M8VP3d3d+Hy+gse+8sorXHzxxf2eDzBhwgRmzZpFe3v7YIc0csgOyJodwmrrc4/JaX6ergmwbgr4yoJMZas0lOyE05WRk85KDRVCIK6/HcrKMzLTeSmjNkNHTuqp2QPB6cqNGUTCyv2TvXPb06qCyaB2jOlmPZZM9jt/y1xT9vdrcxIYtCFoamriwIEDHD58mFQqxSuvvMKcOXP6HdfZ2Ulvby8zZsyw7otEIiSTyjcaDofZtWvXgLGFUYkZkM3Oz3c61WTgcKq/zUmhtALOvkDd70hv1BwOeGCxUqyUEhxOxPWLEOt/CuuehrPO75f7LzxemDYdrXqcHQQeQQghVF1ISVn/ByPhTKAYMt93HjIeU0bCkryot79fm5PCoF1DDoeDb37zm6xatQrDMLjyyiuZOnUqP/vZz2hqarKMwssvv8xFF12Us43t7OzkscceU/5tw+Caa645rQxBfjcxS3EyOzAozEYljaqLmCkL8Z27Yf2dKsBodgZL//BN15pcsHxABc2aslK6eiKjzi10OmPVdbQ3Zzqj6an+rkFDhzPOgpZ31aT/+IPo85fDE2vTcSUXzDwnR/LCxmYwCDlKyxX3799/yt/jZPnuDMNA3v51VSBkqk9amjKaWiXWNaodgtlxzFusVoa6rnYGRSVq4mg447j8wrbPdHg41tjNrDHpcqv/H38I3n4tdwfgcCoNqWhvuj2mUPITnXvTC4UqWPEwWnnlSTUEp/N5H8mcFjECm+MgEk5X/qJWgDPPyjwmDSUUt+MNiEbVhJ9KQm+6+YxIP6c3rDRpbL/wqMaMG2iapiZxM8XUpLRCGQEz1dSsQ9j5toonOZxgSFh5K8aG/rITpmyIYRhKPmR0rvNshhhbYmIIkCVlmTRRhwO+cRusWpjbOQxg9zuqeKy8Ki0toan/zV2E0Gy//+mE2wN1DfB2VhZZfVMmscDjVd/3u2+r7z9lKEMRjahraccbyHhMFQ2Sl6Vm6EihZZrp2C4km6NgG4KTTCFdfy2VxCitUJkeQoP7FmVkI/I7iBk6TJikdgYdLTBtBnx1vno9TTu6AJrNqEIIgViwQmUC7WlVGWXX3Q4r5qkD+iKAVC5B8zpJJjKppfmr/ewsNXMnkdV72cZmIGxD8D4xt+CANTn3aypz3cJMxa/ZXSyVzASES8qUC6CsMi0jnc4c2b0j80N+J6BkIzQNOet8xIJlw/aZbU4+mqapntFmO8wn16tdwjuBdD+CDhVYNolHM38XFVu6VFbcwZQNMXRAQG29rVFkc0xsQ/A+kFJibLjXCuzKWeejmT19zRVZJIxcfiPS3Jpftwi57IaMO8iUkHC6oP4MQKp2hebKL7s7llSiY+xptld3pyEimUCaGWMdzfC9DbDixoy0+PQPwHvb+z/R4VBxgCfX9VORNZwu2LgK9rYhN903cC8KGxvsYPH7w5zwU8l04VZzppHJtCaoTBfUpYu9rJaGdQ2Z10grSlJcAl+8TrkGDD2TKlpWoWIFpRXpmgMX1Nl6M6cbUkq1oq+tV99zJAyPP5iOD6XbW2YbAc2hHgMlWPiDNdC22yowM3WntFRS7TDswjOb48DeEbwfzAl/R7oqOD1Bmzn8RiwKP3xA1QUYOvKeW5RrqKM193VM98+DS1RQuMKnVnXXLVKvl0xYaYaAHR84zcjpSmcuDKTMNLLPRwj4wDmAgJ3blDtxxza1aKjw5XQzs9VlbU4E2xC8T8R1i5Dp9oHZMQIZj6nCn84OpTC5tz29M0iniZpUVKlV4L4OtWqrrFZdycoqMpO9x6t8u+msEJvTDHNnGfL3l6suhJTqetK03GM1ByxbCz9+GLn028iGM9AWLLfVZW2OG9sQnCD9hOTSvtfM/c1qy25WD9fWw7vh3GKykjL4zvdgwmRYcq3VV4DScvsHO5YwV+1SFq4wNikqhcYzVBppX0TVGVRWp4vNVF9jVt6SiTu98zoyHkMbqBeFjU0edozgRMkXkjN9r9b96Zzw9FadPS05PQI4Y5ZqMn/PzXDzF1VxkFS9BWRPyC4AGkOYrkRt1Q9g1Q8zQoJOF8zIKjp0ueDr31ELBjO7bPJUtbCob4KvLuhnROzraGQjpSSWMkbM92QbghOlkJAcKF/+1Hp1/6zzEfd+X/1Ac9RFUR2nsg1DbfoH3ROEO76FseHeEXNx2Jx6rErjomKYdb5KNJh1Ptx2L5z9QbWgaDgjd2UvhHI9hvzKVeT25KrWSglPrLWvoxGKlJL7tnYy77lW7tu6b0R8T7Zr6ATJ7+wFYMSiyCfWwp42mNqAmL8UTdPQXe5MvwHzy+6LZBqYO53w7XSDc1D3tWdSRAsVp9mcfpjfs5ivUj/N71vefGfudTZrdrpGpVG5hDqalevxibUoCfMy1bNaGnYh2QgmrktaAjG6+1RhYCxlHOMZpx7bELwPzGblOVkfvT0ZgbBEHF1KeOwBQMCMWbArnQJo6LDmSVUkVFXdf8dQ12AVCWXHImyZgNOTgWJOkLnOTLS02qx0uVVSwuMPqsVH9rVnSVQ32JlCIxSPQ9BUpb6bpioPXqdG7zCPyTYE74PcvsPNuc3k9RTysQdVv1kzYNyZZ/F/uglx0wrkpvuUETGVResaLGlhGY9lYhEwplZ3pv/U4xCnv/HLjjmZtwf4noUQSLcHmb/4cLpUdXpfxEo6YMFyIN3DwNxh2DvMEYEQgqWX1xLXJR6H+h6G+3q3DcEJkrOCMxuPa45MIE9oKrfbajJP/2yQfe3qPjN1sMKHuPPh3NTRMZoHLqVk8a93svNgmKYqD0svrz29J60T/Z6zU05BGYFZs+GfvgV3zVc7gr5etVDJqziWm+6zd5gjBCEEXqcYMde7bQhOlELCXppDyUU7Xcq1s/Otws91OqGkXP3gS8tzJoAcI0D/WMRY+dHGdcmuwxHLf6pWTVirp9PtPJzw9+z2qKSEkF9de95itej43k25SQj5O0pz4TEGd5gjmULXu9dpG4KRT/YKztwR6CkwUAbhWwth+Q39ff8I1Vj++ttVpogQiGNMAPk+4tMRKWXOJO9xCGaOL8Uw1ArJrcF9WztpCcRO2x3CiXzPGcXSe9SCI9qr6guyd6AOZ0bCGgouPMbKDnMkkX+tA7g1mF5Tgq7rNFV5LFfRUGMbghMkp/2kYShxsFBaSE5KeOKhjBHQHKqR/JRp8O07EN4iJQDW0Yysm462YJlSJx1DZP8YIHeSX3LZFBIGrP7EB9h/uAuPQ/TLsBiuFdNwUVDWXNPQr/8u3PaV3LanoIzAzLNVtXveQkOOwR3mSMFMGc1e0ACsfnE/baEEDVVellw2xY4RjCbMFZwmJca06ZlWg8UlKq/bxDCUkRAamsOBEY8pHZlUEiJvYMSiOIqKh+1zDDXmj6HZH6Wh0sOiSyZnJnkJ927ppD0U58yJXSy6cFx6h0BOhsVwrZiGg4EyxyzjkJ9/7nAqd+V725EbVyEWLM9ZaIyFHeZIpdCCBqDFH6M7msLQdcIJgwrP8Lg/T4oh2LZtG0899RSGYXD11VdzzTXX5Dy+ZcsWnn76aXw+pcr5sY99jKuvvtp67Be/+AUAn/nMZ7jiiitOxpBOKTn673pW/+FYNPdAAZQrITnpcsOjKzMruFQSHnsAefNdY2Z1Ftclzf4o/qhOINbHQy8foLHSDUB9pYe2QAx/VGfX4QhxvQavU/TLsBgr5woomFFkuNzIDfcqZdGiknScylC7z75I5vpqb8Y4cghRMx5NU3WjdtbQ0GPugN1a/wWNlBJdSjQBobjOrb9pY3q1d1jcn4M2BIZh8OSTT7J8+XKqq6tZsmQJc+bMoba2Nue4iy66iG9961s590UiEX7+85+zZs0aABYvXsycOXMoLS0d7LBOGTmrtKn16gcJ6gcpRO5W3eGEFevRyiuV4djXnvti+zrGVNDO4xA0VHoIxPqQEtoDcTZ+qgEhBG4NVr/YSUsgzoxxJRiGQTQp8To1K8NizJGXUSRdbuQj98D219V9TpfahTbMhGsXqsIyc3caDsCy65FllRgP/RghRNZ12wBfuwlRVmEZCZuTT747yHR9mguauK6MgCHVP39Mp7k7RiiuU+FxDKkxGLQhaG5uZuLEiUyYMAFQE/6rr77azxAUYtu2bZx77rnWxH/uueeybds2LrnkksEO69SRv0qra1TGoHYaINTf5gptWhNaeaWV/20JjBm6MhJjLGgnhGD5FbXcu7WT9kCcJp/HmugBll5eSyxl8Ohfu/jqfzUDgtmTill2GgaIj4f8jCIZj8GutzMHpJJKzHD7G+oau3Yh3PrlvNTlELL7iIpVmddt0A+3fx1ZWoG+6gdo3qIxeX5PNfnuoIQBXmfG8Hocguk+L63BBMFokpQB4bjObf/dRn2Vl+WXTxkyQz1oQ+D3+6murrZuV1dXs3v37n7H/eUvf2Hnzp1MmjSJr33ta9TU1PR7rs/nw+/393vuSEFKqQLEhp5RDJ2/TAWHn96Y9uU2wvzlqpeAKUudlqk2f9TS5c6REhhLaJrGiisKu3rMbKpdRyIkDQBJiz825gLE2WT79QdUpNF1SxJd5l9PmgPuvwM5bbpatBiG2i1IlL7V8hsxGs5Q16bNSSW/grhQfOvWiybhKC7n2//3DQJxSUpCd1THH+3l3i37WHHl1CGZI4YkWPzBD36Qiy++GJfLxe9//3s2btzIXXfddUKvsXnzZjZv3gzAmjVrqKmpORVDzcHpdFrvI6UkuHoxyZadyL5ekBLhdOJ8/CFS7e8he8KQTKA5NHxuJ+En15LY9lck4DzvQ1QtfWBot3pZYx+pmBXE5q5ASkk0qTNz/BG62wJIQCKYPL5m1LgwTuV5l1ISPHcOiTf+onadJi4XNdXVCCEIzf4wqdZdOKY2UPK1BYTuXYj0d6F1tlH58E+RXYcI3HUz9IQB5ULS9rVRU1ba73onaxEz0hmp1/u6z9bkXOMmUkoWP7eTv+5RGYfFLgeClGXsJdARTlJa6aPI5Tjl4xy0IfD5fHR3d1u3u7u7raCwSVlZmfX31VdfzU9/+lPruTt27LAe8/v9zJo1q+D7zJ07l7lz51q3u7q6Bjv0Y1JTU2O9j4zHMN7brgp5HE6oqEJOmUay5V11X7rFpJFM0nXLV1R1ZzpekNy9k679nUOaKpo99pGIlJJVW/blpNOtfnE/LYEY02tKcAhISeiJpeg8dASvUxsVAeNTfd6Nb9+hkg72tKaLGQVMm07X/UvVfVMbYXIdxtuvE1z0Tat3gTGpDv83PgGplLp+Vz8OT2+Azj0YtQ109UQY5y2iq6trVOpcjfTrPV9LKJYy2HEwRCKdPVTiNDh3vJf2YAyJwKlpNFa6iQT99KbPfaE6hBNl8uTJBe8f9DKrqamJAwcOcPjwYVKpFK+88gpz5szJOSYQCFh/v/baa1b8YPbs2bz55ptEIhEikQhvvvkms2fPHuyQTgnS5c5Ubhq6+sHNXw7105X09JmzYdEqpf4YDmY0YJyuMRcLOB5iKYNtB3vxR3W2HezjcG+SZn+U7r4UzV29OcJrI1G2d7jQNA3t5jsRq36QjjmhJvc9rcr/v7cl0//a0JXb8vbV8MXr1XGgYghPrVfJCrX1quF99sQyUM8Nm5OGxyFo8nlxaQK3Q9BQ5UZzaDgcDmaOK2bTPzTmZA+d6t/AoHcEDoeDb37zm6xatQrDMLjyyiuZOnUqP/vZz2hqamLOnDn89re/5bXXXsPhcFBaWsq8efMAKC0t5bOf/SxLliwB4HOf+9yIzRgSyQRSS2/RpIR9bapB+PxlyHgM+cMHVHGZicMJdz+qttbpgLFNPgKQpAxY8rsODAS+IgcfmFBGIpGgxR+jyadcE2O5qCwfIYRyIbz7llpwvBeBmWerB+uaIBlXixFQLqQHFqv7s9m9MxPrSsSRZn0CjFmdq5PJsVbvQgiWpZMjqqqq+PxTrxFKpN19/ph1jMmpLqw8KTGCCy64gAsuuCDnvi984QvW31/60pf40pe+VPC5V111FVddddXJGMapxe1Rwbad/RvWS1Bqo9l4i+CuBSAEctZs5ILltgJkFl6nxuxJxTT7Y0TiOv6YQXWRk3Ufr6dpygS6urqIJvUBc7BtskhLUIuVm1R20b4OWPmdzOMhP+wFyqtUoBiUASivgtppyMcfQu5pJXjGmcjrbh+zOlcni/zCyeVX1BaMcQkhKHI5iOsyYwSAnrjOA3/sZEXW844n8DwY7MriAlgFY+kVkpQSIxZNVw+XQl2jaiKStW1D17NeISNJDVjba+n2jDrf66lCCMHSy6aw8g/7eOtQHy5N0OjzWPnTUkpufK6NUEyn3KPxo2saSaGN+BjBUCE8XuSZ58I7r2ea2oMqNtv+Ru7BldVqEdP6nvlsOPM8tSHL6meQat1l1bXYVcjvn/zCyZVb9vHdS6f0CxiDmjse2Nyc3hsrkga8cbDPyhoyXzO/DuFkMjpSMYYQM1BmLL+R4Oo7MAwDY8O9Kj/7nb+pFdW+dpX+mUYIoZRFM6+i1B6FyI0R2L7XHBIGtIfipCQkDUkiZRBNqlTIcMIgFNORQChucNcfOnFiENflmI4RmAghVIzqzPPUyr5+ujovO9/MxLJKyuD+H6mdwrULlUAdqN4XX7tJGQ9TzrrCh7Nxpu0GOgmYhZOgbPTbh/r6+fbNjLlYymDXkQgSKHfB2eO91vNaAzGiSd2KDax+sfOULYTsHUE+WZN1qnVXRr7XXN0LAVPr1Reazt0WHi9y1mzVhyC7srikHO76V7SKqtyiMrB9ryjlxanlbvxRJc3x5qEo//KfzfxdQzcLP1JDmVsQTqgfzjuHo3zh/2umwuugyec9pauj0YCUEr6/GvbvhWmNiHlL1a4z+6BoH6y8Fdk4QxkC8zxpWq46aV0T4rqFVE6pzckAtHl/KP//FL7xy1ZCcZ2UAf6oTksgbsmqm66jxioPhtnFVocDPUkqPBq9CYNIQvLgS/vpCCbojp7a+JhtCPLJmqydjTNJlJSpvrC9PcpMn3kuSJDLb0ROa0JL+/61BcuV++ixB5QscCqlVmBPb7S6Rdm+1wymH3XHkWjOttgAdh4ME074mFlTzOsHekln2FnFNvjjlkDd6SpNfUzyKtxFMqEWGkXF0BNSxxg6REIZV5HZ83haE5q3SBVDZl2LY+4cnkKSUuDU1Pl0alDmcVi+/VjK4I0DvSQNSTjWR2WxCwDdUNe3r0glpSQNyVsH+zhvUgmIUxsfsw1BHtmTdcWkyRy55zbo7IAPnAPXf1cdZMr/bg9jhALWit9RVIy8+S6MUAAWX6t2ETu2IeMxhLfIen3b95rxoyYL9O1OGZJbftNKJCEtI2DiK3JQX5URqDNfayxlEUmZdo/V5e0u4zGVvlwA1ccg0/PYMgBZ16JVCT/GFyknA5Ue6oEANFa6WXjJlJzCSdNFZEho8BWj6z2qLEQTNFR6efOgcuMJAYsunoSmndr4mG0ICmBO1iJv1aVpmvoSzQP1FNzxLYyzLkC7aXlmVWVP9MfEJSR6ek8swCogK3MLokmjoIFwClj39/VUeByWQN1YyyLKKfaqa0Ss3KR6D5jZa5qWiRGYFJVkMtzSPY9lAWnr4OrFGLt3jPlEhpPBQKq5UkrWv3JQeemk2gG/uT/MOeOLEUhag3GEkJwzoYiOYIImn4ci16kXoLMNQR7Z6Z39ujy5Pco9NGMW7Einixo67HgDIxRQq363B55clwkUzzx7zMcC8pFSsurF/fSk/f+VHsG0Cg8d4QTTKtzs6IqTr6zjFDB7YhGVXqctTZ3tEkovPqxV5hlnwe7t6jq1BA4dEAkjyyr6PV/GYyrFWUpSre/arSxPIvmquVJKgrEUzd25O+GELmkNxhCoWEJPPEqpC6ZWFrH40smnj9bQaCG/tJ671uf49AHVALxzb+4T9RR895tITVNZHPvaM5XF+zpUVzJ7hQWocxyK67QHYkiUvZxWVURHMEYgZiBlnAtqK/jb3hBJI2MMyr0OFl2aiQXY0tRYixPrujVTSTVNXZNOF5RVgDSQ99yCrJ8ONy5RMa+0a8msIaCuEVfjTBK6YScynESy+xHct7WTbQd7SRng0qDIKYimVMLJdJ833dckTk9cJxCHwKE+vvHLVn78mSYcjlOrN2QbgmzyJabjsRxXjx7tg7bdmaKcbKShoj0dzSpnW0oVYA75M6miY3yFlV1oY0ioLnJQX+Vl6aWT+MYvWwAIxSW6YVDigmBc/WDKPA6m+7w5Er5jlUIJBzIeg/bdmcw2w4AKH0ybDl+dDytvVddhezNsXKUWKnUN8OUbYdVtVqvVik0/o9vvt2MEJ8hAVcTZ/QjqK9y0BeLWTqDErbHuY9MApdhs9IURQgWS17zYybaDfQCEEwb3/GEfd19dd0q/E9sQZGNWD0up/vd4IaKCNnoqBUuvg0i6slhouQqQJhKl4TK1QS1397bZK6w0Zpm8P50Zsfbv6/E6NRKGpC+Zzq8GXtsXptyt4SvSaKrycNvFk/v9wMakWyhNvziU26O0r0J/U7cdDli+znJVymlN0AFMmaqux2C3Sotecp0qhHQ4VQqp6dpMFz+OxXN7ohTqRWyet2xZCCkl9ZUeAgf7kEAkbvDtZ9tJGRKH1sa544tYceVUilwO7rxiCl//RQvhdLVxRyh+yhMibEOQjxAqekm6lVy0T7WgvOc7GSMAqjNU/XQ16Ru60n2fMk1lGIX8Vsm/EMJeYaXJLpNvrHSz6S+HaAnEmFbhzjlOSnAIwbq/r8fjEKx7+QCtgTj1VR6WXTaZNX88QLM/SpPPO2ab1mQjhEDctAL56ErY26p2Ak9vVC6fqY1qwdLbA7u2q8r4Cp+6be4gSsoQ1y0EsCvfT5CjaQB5HIKGCheGYaj/sxJNUhKrxW3KgNcPRrl3q5KVcDgc/PgzTdzzh310hOKcUV10yhMibEOQjeUa8gMtBO5dBG/8eYCDBVx3Ozz+kFplTW1Q9QLfX239kMQo0XIfKrKDvFJK5j3Xij+qE+xLoWnqYixxazgdmmUomv1ReuI6SQP8sRTf+8M+dhyJkTQkPQf6iKWMIdFrH+kIIVSMCqFW+Z0dauXfE1ILFbMau7cH1jwBP3lU1buAda0SjxUMJNsLmYEZSAPI7K3R7E8Qihs0BxKIvIp4lyZIGRnj0B6IE0sZCCHwODTuvrpuyHa+tiHIJjsQN7VeVRYPRG+PUhuN9KgV155WS43ULhgbGDPIaxgGhlSbLx0VXqnyamz6VCPjxo2jq6uL+b9uwx/VcWkq105K6AjGIKd9hw2grrk9rRnJiLoGNflHwhkjAGpi/8kGpTHkLU4LwBoq4yin2rgR+cTafmmmNrkUymAz3UXvdfUSjKtzH44bnDPeS/CwUhYtc8G/fkLteB99zc97h3uYVulm7cv7aQ1kCiWHKi5mR9+yMANx2r3fRyxYgbNhRvaDcOe/qkwMAKRabZlxgkgYw+myJCfsH83RSRjK/ZM9lRsSa0VlrrSqi53MnlhMuds8n4JzJxRR5dU4b2KxHUA2MRcxldXKZTl/OUytV485nFBcpv41fUDtYMMBVXXcE4J3XkduWAmQuf6vW2TrYh0nanGj5cQGmv1RywgAlLsFt1w0iSqvul5jOiz6nz08/OdDaEA4pvPmgT62Heijuy9lyVEMFfaOIE2OPLRHpXJZ/QcASsthch3MOl9lBunp8n0TQ1c/qkpf/xe3schOp2vyeTC6DYIxAwn0JQ3u3dLJ3kgH9eVOK0hsupFAogmBlGrlZZvaDPnZRCTiGPs6VBygvEq5Lre/Ae+9owxDWaXqWWyypw3j8AGkcKKZ17+ti/W+MEXnArE+pFR1Mg2+Im7/nz1IBFVejUhCqsp4fwwhlPAigIbEV+QY8kJJ2xDQv35AzFuqVvht2a4hobbZ37pV/dh+/Eiu0qPNMcnPsFhy2RSCsRTX/6o1nVYnaEvXEwT7Esx7rpX6CjcIQU/cwKUJGnxeS15CiMSYk5c4GjmN7s0dgpQwpU7tAsz9l56CxffDfzwO76YLIyNhum74JyirwHjox6oTWrrpks2JIYRg+RW13Lu1k/ZAjJQh2XZAZQtVFzlY+/cNbPzLQVoDcRqrPLhcLl7dEwIk500oYtGltQUlq08ltiEgHRTraE4HiVHZF/vacyf53p6MfpDDqXK1sx8vKUOUVw7puEcbhTIsNv3lEKACZ7MnFiOlQVsoSbAvRSBmEIhlJqJKNyy8aCLrXzkAxGmsdI8peYkTQQiBmLdU9SfoaFGLmGzKKhA3LUf2hFRW3PIb1P09QWQ4aO1s5eMPQUdLjsCizbHRNI0VV9QSiuvc8ptWVTwJ1Feqepgll06mJykpd2uMGzeOfQcPI9NqxkNtBMA2BKo0//GHVNDX6VJ+1T1tKujmUqtRq1TfnPizpaZNGmbYP5JjkJ9hAdAaVEU2Pq/GvI9MUIZBCIrdwpKgMOmJS9a9vB9hdnuyz/dREckE0owHZHPW+Uo63dwFT5kG2RqwP34E+Z271ALJlFbf0ZMjnmhzbIQQVKSLId84oArEdnf1ccOvmlWKtEOjyedl/WfH4XVqA9YjDAVj3hBY2RappMqvvu52pRXU3gx9EWUEzC/kaA1R9rWr1VVZhW0QBiA/wwLSBkGCLiW3/qaNYHrydwolRJcdL9OB1qBqCOSP6ojAqS+0GdVku4ci4ayeGo7cST4ShhlnqfgBqN2wHRx+X+QXOwohWHjJFCtVOpzMnkN0eg700ZdInfKexMfipBiCbdu28dRTT2EYBldffTXXXHNNzuO//vWv+X//7//hcDgoLy/nxhtvZNy4cYDqbVxXVwdATU0Nd9xxx8kY0vGTp91i6rTLeAzXTx4lsXunChJ3tEBv3va6slr9uDQHGLql52Kn2g1MvkbQ0ssz2+dA1g4gJaFQdUB9uROH04kYg8qjJ4oZQDbCQbj75kxyw55m1bTGNAx6Cg7sg/JKQKisI7dH7RFmna+Or5ueI1lt05+BehV7nRrTfV6au2P4Y7kxxaQhufO3u7jjovE5u2W3BrGUMWTV84M2BIZh8OSTT7J8+XKqq6tZsmQJc+bMoba21jqmvr6eNWvW4PF4+N3vfsdPf/pTbr31VgDcbjcPPvjgYIfxvimo3ZL21VUsXkPX/k7lOor19X/y7asRNeOVsuM9t9i6Qu8Dc/vcWOXl9XRAzUQnx2EBgCEcrBjj3clOmKc3ZtpUAoSD8L2bVazLdHP2BJUrdPXjaOWV1nnVFth1McdLfq9is1IY4NaLJhFN6tz62z2E43rONb37SISEMd7aLbuETAea4zT5hsZNNGhD0NzczMSJE5kwYQIAF110Ea+++mqOITj77LOtv8844wz++Mc/DvZtTyrZ2RaGYagA2942Qk0zkF/7Tm6rysyToKwcTdMwSstVeh7YqXbvg+wsi9ZAjEjCIKlLhIDZE4po8cczuiuBGHFd2tXEx4tZLa+nyDGrfRElNyFl5tpOJftN+HZ/jeMnP23UrBRe+9J+th3sVfkl9C+DrKsswjAMcDjwOAQr/9DJ6wd7lSfaPzRuokFX4/j9fqqrq63b1dXV+P3+AY9/4YUXmD17tnU7mUyyePFili1bxl//+tfBDmdQSClVxtA7f4NgN4nX/wI/vF9lCAmhxLycLvXvnDlo3iL1nE33WYqOYt5Se+V0gph+VZHW0C91O6gucnDBpBJWXFHLD/6hgXPHe3EKiCQla1/ebzewP15MIUWni35TUF8kd4HjdOY0UJHxmH2eTwBzQXPBpBKqi5yqQxnQEoiRNNQOt9DZfOdgD1/9rxZWbd1HLGXQFoxZocn6IXJ/Dmmw+MUXX6S1tZW7777bum/Tpk34fD4OHTrEPffcQ11dHRMnTuz33M2bN7N582YA1qxZQ01NzUkfn4xFOdzZngkKSwP270E4HcpdVF6J76Gn0IqKEd4i5UaKRena14YR7EZzaNSUl42IzAqn03lKztHJxjAMvvvcTnYfiRCOpUjoknElDn70pQuo9DpZ/OudvHuoB92QGIBhSNpDSUorfdauQEpJLGUMS9pdPiPxvBuLV9O94IsY/q6jH+gtoWbyFIQQBFcvJtX6Ls7GmVQuuX/Yz+uxGEnn/V8/N87y78dSBjPHH+bPHSEkKglCopIgnJrKSFQ1NJK2YILq6mpmTerm3UM9nDGulAc+dSaaduqr5wdtCHw+H93d3dbt7u5ufL7+1bVvvfUWv/zlL7n77rtxuVw5zweYMGECs2bNor29vaAhmDt3LnPnzrVud3Ud46J+HxiGoSqGQa3+yyqgtgHZ3gyADPjpfvgetJtWIHpVzEBKiVHbALqBUdtAV08EEekd6C2GjJqamlNyjk4mUkpW/mGftQ12aQJfkYMZ40vpDQXoCRj8tSOQ081JCKgrdxEJ+unN0nUZrrS7fEbieZdSYkxpgGQy09i+ENEIXfs7EUKolpXBbhK6weF3dyCqx6nzOkLjBSPtvEspWbJln6WuW+HVCMYMSt3Ckp5IGZJSF8QNZR0aKt30hgIsunAccb0Gj0Mc1bvyfpg8eXLB+wdtCJqamjhw4ACHDx/G5/PxyiuvcPPNN+cc09bWxuOPP87SpUupqKiw7o9EIng8qrIuHA6za9cuPv3pTw92SO8bkUwgRdr6llYwbv1POBJOi8sBIAsGg8W1SsLX1hg6MeK6pDUQtTZg504sZtHFk9j4mp/5z7WRMjK9iwVQ6dFo9HlZfvmUgprv5m07nTQXKaVqXHM0IwAg1bEiW3wuEoKl1yHLKpCNM22J6uMkljLYdrBXqeZGoziFWuhMq/QSPBS1jtME/PzrHyTaE8rZ0Xoc6jWAIdnpDtoQOBwOvvnNb7Jq1SoMw+DKK69k6tSp/OxnP6OpqYk5c+bw05/+lFgsxrp164BMmmhnZyePPfaYCrgaBtdcc01OkHmokS63cgcJAdJQQeCbv5yryWIYGE4XDgpIU8xfNlxDH5W4NZAozaByj8ayyyaTMODdwz10RzO+awGcN7GYOy6d3K+Rd36PA7N371iepHJ0s0RaGiVyDCMAqmDysQeQN92p0k67DqtmTKCMSHtzRt3Uzow7KtJy+ShSUhVNfufvJvKtZ9qsWEFPAh54oYXb/258Tnxm1ZZ9bDvYCwhmTyo+5X03TkqM4IILLuCCCy7Iue8LX/iC9feKFSsKPm/mzJmsXbv2ZAzhpCCSCSU0JyUIDdnZkWsEQNUSPLoS44Z0vUN2a0v7x3FCZCuQOjWNhAFrX9pPKJayBOVk+t87h/pY98oBll2eu1Awi9RiKYN1Lx9g/q/bRoSLaLjIX5xo85cpwcSyCjWZe0sgGhnoyUqJ9NGViJvvRFSPQ5ZXqueVlqtdwh6gzs6MOx6yU58r3Ko+wOvUKHMLwolMR75dhyPE9XHWTtbc5ZqxgxZ/zO5QNhRYGRKGoSQmpARpELjn1kIHw663kStuVD+IKdPUfVmNxEeqH3Wk4XEIlVkRUD8SKdM/gALyuymJ9YPwOMipTjblem0XEf37bifiaB4vxoM/VvUuqRTc8c3c5zicuc1rOlqQ8RiatyjzvJIy2HRfemazM4kKkV9V7BBYqqINvmJAsOA37ZYRAOUuOmNcKS4hrQCzucvtifcBgiaf1+5QdqqRUmJsuFdJ9Oq6ChLPPBv270XmZ1loDpV7HetTAnWRHnV7WqNSLMVu9XciZK/mpZSsfWk/PXEdlwbFLo2+hEF2RX5juuLSDA43puUpWoNKxbGxUrW8HNMVx3mV8ubK3awHkCUuKK1QrqKSMlj6kLqGF34VZDpRItqLfPwhjOsWqbhXeSUyHsMwG9/sEWNy93u0XtmFlHXPnVjM62mNoe2H+yhzO3IqizXgrHFF7D4S4ZvPhNGEunYXXjLFKi6DURIjGPXkFNyg/t/XAbX16qLPzqMuq4Bla2HlLWq7nEoqQa+db6nXEcJ2Fb0P1r2sehCHYzopCdXFTh7633Xc9nwHgWjmhyMRRJM6u/1RAlFdZXkBgZj6f+MnG9Jt/sZuxfFAlfLWAqWuCVb9QD3+003w4FJ1rWdf56kk7NiGXH6jJZkykIEZKxwrOy0/aSFhwO2XTOar/9VsxQrqqzwQjJPSJQ4haajy0hGK0x3VLTdST7yPec+1Mt3nHVL3pm0IzAt8e1qUSwglOS0lfOBc1XPAZMo0eHpDf0nf/NeCMfljeT+YPyB/1oQfiqZU45oqD2/GoyQNNUm1dvdx3S9a6E3/sIIxg3KPhs/rsPyvY9UAZNOvGjjbXRQJw53zVSX83rTKbnanPYcTSsugN6Iea29WqqMe75jOjjtWdlrB3sUOB7MnFtPsVymkuq5b13m5C5ZeNplrn21HEyp7qMylEUmqhjUtQyyoOOYNgRACbcFyFSOIReGeW1SAeOc2cOadnmQcOg9mFElLypS7KC3SRSKuKouTCTtGcJyYPyDDMKyVfUrCgv/uYLrPy4//sYH7/7iftw/HCMRzfdMS1Qv23PFellw2xT7fWWTHqjILFKncmSG/coMK1HWsZ4wwegpCQSgpVbd7e5CPP6SiAntax2x23EBN6k0K9S4GJaq4css+3j7Ul5NFFE5Cd8xAE6pFa4VHY/3HMw1r7A5lw4AQAuEtwnB7kNmBsFSevtDunco4OF1w5rkwfzlaKol0uZGb7rMafQs7NnDcmD+gQDTJN37Zat0fiOq0+mOsffkgHYHYgOFJCbxzOK0/ZLcvBgpnDmnzl2H09cJtX1EHRXvVQkZKtQuALLkJqXbFTpdyE+1pUSd6DKeODjTR5x9jJjK4NUnCUN9FeyCeYwRMXOldr8ORoqHCRaXXybJjvMepwjYE2STiIArMJk5nui48pX4YDidcdzsOh0MFl+MxZQTs2MD7QghBVZGLOZOL2d3VR18KUob69+bBPvKbgZY5oa7SxfYupZypA9GkbruGTApkDuH2wOMPZpor6SnlJhICikrgjjU4fv4U+luvZdxEoDqV1U0HpAoSj2GXZ76EejamzMnal/bTEohhSJUa3ejzMK3Sjf9gpois3C3QNI3b/2cvjT4P//6VC+gNBaxrdziy3WxDkMbqVNYXURO91cRDg/seh0QC7l6gDIHIs9Z2bGDQKMGuqYTiOguf76CrN4kmZD8jUOqEx/5xOgBf/a8WkoZEAAuf7xjyANuIpcD1KOMx2Nuee5xpFCJhuHMeenmliovt3q7uP3M24vpFmT4Edlp0QcxA8u7uPnriBimZVUPgl9x5xRTe+O0eQGUKPfixepb+fi/d0RTSr/a6w31ObUOQRvUtblETfVmlWhX19kBxMfxkgwqsFZWApsG0TJMO0xdrxwYGj9mb4AMTyth5MEx9pYdt+3tzjIHToVZTHofqcfzq/l4kqmPZUBTejAbyM4cA5BNrM4scb5Fa4GgCECrzTaL6FPSElYHQHHDdQtWoycTe5RbE7ENgxrgAq0dxMKbznbQRACjzaIwrdtJY5SEc1+mJ69z9/K6cyuLhwDYEpHcDT6zN+EVr62H/HuU/7Y0oWWqACh/izocR6XaUVg2CGRuwm3sPGiEEaz55Jp2HjuASkpVb9vFG1rbaLDwDwfwLJ/L1X7RYj02t8NgSE2myM4dyFjmVPrhnE5qmqdhWPAbLvp3OHJKZWgKjgFPbpiBWH4JobmMlU2U0m76kQTCWYldXJhsuv7J4OLDDa5DxqaaS6t+7bymjkI0Qqt9AVk9iq+9ryK/yruOxoR/7aYLpYzUncbcGq7bupz0Y5/wJXn50TQP//rkmNM3Bjc+2cM8LeyhzCSq9Soq63C1waYL5v27jvq37bB39bExXUWU1TJuO5i2yUkCFEFDb0P85Wb0JbI7Noksmc8GkEo41l6cMuOU3bVajJYDG6pJhL4C0dwSgfihT6jLBNZmOVGYjNJhnZwOdCkyRrZZAjMZKN/d9upJ7/rA3ayeQoNitLlVT0TFwMMrKrft58tMNRFJKwG7Br9uVWJ0cwxITBch2FUmXGxJxjHSmG+3NWYseAWXlyi2UjjHYu6uBMRcvZkFkfYWbn3y2CSklD758kD3BOI0+L/9yThU/fqObjlCcYNwgnMy8RoVH46F/OJNAIDB8HwR7RwCkt9E33amEtUycLrRz5qidAIA0EH25Yl3C44VZ56vt9qzz7ebe7xNTstcf1fnbgSj//JPX2ZblDookDNa+vD9dSZyZlLYd7OO+F/dT4XHgdWroUrW31KXEbV/ZOQghVNB4030Yy29EblgJ77yerilIQVklrgsuhAd/jLj3+wDIFfMwNq6yd1cFMAPE855r5W+dEfxRnTcORnngpQNomsZdV9ay8VMN7DrSy63/s483D0f7uYnOHl/EU//YpLIPhxl7R5DG4XCgr/qh6j0QDqofh6ZldFnKKqC0vJ+onN3c+2Sh8iwk0N2XzHkkaUCrP8b9L6oWldmqjm2BOMFYKv0KEilVDDRhgNc2Brlkp5VmT+4OJ9z5MFVNZ9Dd3Y1MCeSeVmuHLOMxtSCyr3GLQhXxErU4+crPmzl/cgnzPzKRnqxLOZIwcq7dAz0JEoZKfR7unZdtCNJIKeHJddDXa96B3NsKq36gsoFKy1XnpqzgsGYGh+2dwKDwOjVmTyqmuTuKROByaAT6kqSkau1X7tFI6pI3D6tdQoUb6quK2BOMkzIMvvWLFlXo7VIdzqYfp1rj0UTETkuy00rrGlXB5LtvKaPw9Ea4++ECxzWpyuJ0VbEtpKgwK42llJZGlklKwrYDfei6TqkTIulM9HKPAyF1ggm17Kmv8rL2pf20hztoqHANa+qzbQhIZw2ZjTdSSRUPKCvH2TgT3VuEKCoGwIhFVXA4lYQdPUqDZQT0Jx7tCCGsikq3Bs7SSr769GsE0q39plZ6eetgn3V8k6+IZZdP4Xtb9vPWofT9EvpSksc/WU+Fx5FT3Xk8apFjof7AjBXIeAz52IOwrz0tMZGCPa3IWFSt/t2eTExBSuSKeXaxZB5CCJZcNoVQXGfjXw7S0h0jqevWDiBpSG54rh0QzJ5YxM0XTqDS62TNH/fT7I9RX+FmwUcmsPD5DvxpAcXhjGuNeUNgleO3N0NfOmimaSDsKtWhJLtqs6rIxRnVRbQE4kwrd9EezEhMqEMk9764n7cP9eW/Ch6HYPWL+2nxx9ClRBMULDQbqy0uRboREDvfVAsaBJRXwdQGQmtXYLS8l1n5e7yqR4epTmoXS1pIKdV1lk5w2PQPjbg1uHdrJ22BGOGYYTWW2ROKU+Jx4XBoOU2UFj2/B0NCTYmLxkr3sGYOjXlDYPlNQ/5MYNjQIeQn1bpLrZ4gEzA7czbsbYG66XZw+BSR3adg7cv76UlIXJqai1IS3joUQ5eZop1Sl8DpEDRUqe+jJRCjO5pSHUclNPtjxFIGRa5MUO5YImJjBqcTVqyHf3uUxOt/URlz6UZNeLwqs2hvm0qdnrfUXhylyV5ImC5Gr9PBnVdOJZrUuf/FfWw7pNLJDSOTvGCm7JrXaHWRk6e+dD56b8iOEQwrpj803ZUMoan/NQfOxpkkHn8o07TG6YRZsxErvz8mpXiHEvMH0xqIkzQkviIH9RVu2kMJwjHdMgLnTSzGKeDNQ1HePtTHmq17mVruBinRDUkkYdATVwYlu+/r8YiIna4Ijxc563zoaIa6BvUb2Nua0Rjq7UE+9hDyyzfC9tdVLGFnj1o0jVFXaH48yeMQNFa6MQwD3ZDc+t9tNPm8LL1sCutfOciecBJnukOZpgniukQYuR3IQC1CqopcdPfZdQTDjqmzjtuj+ha73IhkggqfjyPzPp/RHUorMVqFODanlPwfzJLLphDXJWtfVq6fhkoPiy6ZzPxft1lVmm8ejgOqwOz7n2rgpv9WPtjWAvruRxMRO50RQiDmL0U+uhL2tMHyGzJJEqCu8+2v57a0TCXHbBppoXgSgEgLVIbjhmoqc6CPYCxlZRO5NIHPq9FU5WHdywdynm/ueEcKJ8UQbNu2jaeeegrDMLj66qu55pprch5PJpNs2LCB1tZWysrKuOWWWxg/fjwAv/zlL3nhhRfQNI1vfOMbzJ49+2QM6bjI79wkrlsIHi9aOhNIeIuU8mLP65kdQd102086RBRatRdpsPSyKYQTBuVuFcdp8nnpOdCbq/eekCSlYLrPawXn7NqCDCKZQO5rz7hEjzXJF1LlHQNIKQnFdVr8UbqjulWsCKqHdra+UNKQPPKnAzRWupFSUl/pYdHFk0gYsPC3HarYkUy/bdM4nDmxm0UXjhvWxeWgv13DMHjyySdZunQp69ev5+WXX2bfvn05x7zwwguUlJTw6KOP8olPfIL/83/+DwD79u3jlVdeYd26dSxbtownn3zSaj84JGTnVe94A7n8xpwCGrNOQDz8f+CR/0Cs/6m6XSADRcZjY3bFdCpRq/ZM4N4M0t3223ZWv9gJwLLLa3nk49NynlfqEpS7NRZfOplplV7aAjHu3bJvaK+vkYzbo1JIK3yqkLK8SulsCaFqZs48j+ziPTQNnlg3pq5xcydw62/aCMVVO0mzWNGtqb/zp+63D0e55e8mqjaUwTjzft3Orb9pI2VIqosclpGIpQwrxqC0hob3vA56R9Dc3MzEiROZMGECABdddBGvvvoqtbW11jGvvfYa//RP/wTAhRdeyI9+9COklLz66qtcdNFFuFwuxo8fz8SJE2lubmbGjBmDHVZB8ovBCnZu6mixCmiyDcJAMYFCTUBst9GpIyfbR0IwlsLjEDz5tyNWsY6Gyi5a/cf96LrkjXTqaeBAH/du7WTFFQOnio6p2gIh1FzfMANx3SKqJ02mu6MtUzMTiyoxuvsXK4XSPS1jKn3UKhrLajgvkIQTytevAfnTd8qAB146wL5wMqcXMcB5470glB5WY5WHxko3ADPHlw57ssKgDYHf76e6utq6XV1dze7duwc8xuFwUFxcTE9PD36/nzPOOMM6zufz4ff7C77P5s2b2bx5MwBr1qyhpqbmhMYppSS4ejGp1ndx1k+nYumDSoHxrvXIWJTgurvQ297D2TgT+W+PorftItQ4EyeQat2Fs3EmlUvu778biEXp2teGEexGc2jUlJWOiNoCp9N5wudopDDQ2KWURJM6H5hwhF2HIqQMg2t/2YLQBCldVSW7NEGZR8Mf1WkLJnJWsBLYE05SWunLySDKfv3Fv97JrsMRZo4vZc0nzzxhYzBaznvmuvWjORzU1NTg8ngY19AE6YwhU2E3OPOszG9g8shsCXqyz7thGASiSWaOL2V3Vx+6IVXlOoJF/7OHGeNKmDmxnL90BDDyrMHecILp48oQ3b3oBgSjSSSwL5JCCJ3uvhSapvHvX7kAIQSlXtXPeDgZNcHiuXPnMnfuXOt2V1fXCT1fxmMYu3dAsJuEv4sjd30H7eY7My6H678LiThJKZHLb4SQn7hhKDneUIBESqdrf6fKuMjuBwsYtQ2gGxi1DXT1RBCR3qMNZUioqak54XM0Uig09uyAXX2Fi3gyRU8y/QvM2VZL6ivcCJGkoVJlDxmGgSHBqQkaKlxEgn56C0xmsZTBzoNhuvtSGEaYzkNH8DpPzHs6Ws67lLLfdVvj8XL4e7f2290a1y6CSJhkaTnd3d3DPfSCnMzzbhgG3/hlC6G4ToXHwaZP1iOEyvxZ+Nt2jkR0DF1n1dypxONx2gIxIglVNyAA3ZC0HOmhvsrL0ksncd+L+2kLxtUOQAgMw6CuzEkk6EfTNIpcQ3fNTJ48ueD9gzYEPp8v5+Lo7u7G5/MVPKa6uhpd1+nr66OsrKzfc/1+f7/nnjTcHphar9w/Uqrc6KxtrikVIQ0jnUYqlBHQ039LQyk3DtAP1tYbOrXEdUmLX+Veh2OpnMCwI+3hSEkodmksumSy1bzG9L16HGLAKmOTsVRbULB5TSjQr8WlTAvVmde7HAOuz3DCIBTXkRJCMZ0H/niAvT0JGirdajoAuqM6NzzXTrlb8PDH6/n+q4dp9cepq/TQEYipwLKIk0JjxZVTLXejlJJ7t3bSHoiz+sVOKwNpuBm0IWhqauLAgQMcPnwYn8/HK6+8ws0335xzzAc/+EG2bNnCjBkz+POf/8xZZ52FEII5c+bwyCOP8MlPfpJAIMCBAweYPn36YIdUECEEYsEKlTK3tw3qM9k/ObGDZAKpOUBKNIcDg3RGheZQmkPQ78ciPN4x4zcdLszgHNC/EbhZVAD0JSULftOugnJotAVjNPm8LLu8NidVtFAsYKzVFliLn/TipmtvqyqmrKzOVBEX6n98ml/r5W6NCo+DUFyn3O2gI6Syg5QwnMyJC4QTktue38N0n4cNn6wHYN0rBxCBuLWYyE5TjuuS9mBcZRAFGPYgscmgDYHD4eCb3/wmq1atwjAMrrzySqZOncrPfvYzmpqamDNnDldddRUbNmzgpptuorS0lFtuuQWAqVOn8nd/93fcdtttaJrGt771LTTt1KWpaZqGvPnOnNV7frBXzFtqCW45mz5AIh7v37Tb7k885CQMFagzcWnpDRvqnxOo8Ah6ExJ/VCcci1pCYD0H+ogmdTRNw62pH9/al/bTGoz30xkak7UF6cleBrtVF74V6zNd+MZgP25N03jqH5sIJwzKXII1f9xPSyBOY5UHpKS5O0Y4YVgeyWBM582DUR54sZP2UJz6CjcbPlFPkcvRbzExUnedQo7SfLD9+/eflNeR8RjG8hvViqeyGrFyk/XYuCm1yneX5/bpl300AhktvupCDBQjWPmHvbx+sA8pwVfkoLbMyY4jcesYPS1BXeaCSBLLEDgFnDephPZAPJ3yJy2fbnWxk02fajzhWMCJjH2kYy6GtH1tGLUN/TLfxvr1nr17zLh2Ykwp97DjcB8pSY4ECsAFk4q588qpA2YaZu86h/KaOWUxglFPnjSvfGKttTvgrvUFZaZt6emhRwjB8itqrR9hyoC9oSTnTSrhug+O48bn2q0esSkDZk8sZlvaaMwa56E97bc10/lcmsBXpI2oVdlwYcYLaspKVbJD3uQ1Vq/37AnbXChkXDs6EOcDNR72huJMq/Syoytm5Yq2+WNWsLnQ+Rxpu84xbwhy2vjlS+7aPYhHDFIqSelll03mni2dbEvXBrQFYlR4nZS6hJVFFEnB/Asn4nVqPPSSUohECkv7BaDICWs/No1Kr92bF9IxNG+RlfE2GnYBp5KBZMrdGtRXKhdRKKanDQK8dVjNFQKVvCCB2/67nSbf6JA4H/OGALJWPFIis/2hHi+yJzKmfxAjgdzUUTd7Q8odJAQ0VHoocjn4wT808OX/arXixuYK7vUDfXkS1oqehOS237Yz3edl2RWFt/BjFbtIsrBMuVuT3Lulk+buXnoSUKhGvcIjuP9/17P093tzAsIjbQeQj20IsiiUUjfWfxAjgXzJXyU3Hae+ysOyyyYT1yXFbidzJpew2x+jMe3uORRJ5FZ+ZpV5SiAQM9h2sK+fRPWYZwxmCuWTHdRtrPJgGAb3/vEArx/sHVCWySngjJpixpe4aPJ5IDCyAsJHwzYEeeT4Q+OxMf+DGAkUUiFNGCql1GoOkm4bGInrvHmgj6//1+6cfrEAsyeW0BqIUVfuYfuRvnQa6sj/kQ45YyhTaCBJkeyeGA/9UTWp74kbBY3AueO93HHpZBwOh6WLNdrSkG1DcDQ83jHzgxjJFPpheTVVCdzsj1qt/iATAwjnGYEyB8z/yAQ8DkE0qXPb83sIxw1K3NqoWLENJfk749Ewkb0fjqddaSyldo3mdVXmhKhOTo/it4/EWPvyAb57WW3BAPNowDYER2Gs/CBGA4UyLdwaGDJd+I3A0DNe2+ym4QA9Olz/qzYMKXMUKUQ6CO0dPb/ZIWEsZAplLyQg15dvGond/mjO9dKn999DSglvHYpy47MtSAQOIUZNkNjEvvyPwdGUR22Gl4QBDiFU4bcAR3plX+XV+NdP1Pe7uJNGrhEAFfCz+xSMPaRURYU9cdVAprHKgzu9yzRlot840EsgmisGp8vc3QBAhVvNE4GYkqbojqZoSTdCGi3YOwKbUYvHIaygnFn12RpM0FDhYuNfDqtUPqDYLehJFP5RCnJ3BOYkAOT0QbA5vTATEJIGVHkFt100yYo3NVV5uPWiSZiZBQ4BRU6VnuzUlARFOGGgG1DucbDuY1PZ9NcjtAViGKjFyWgJEpvYhsBm1JIfOwC1onvwpf28cbAXidodbPxkAw+9pBQgm3xebvzQeG75bQc9CSUr7M4yAvdt7eSNA72AZPbEYju19DQl360I5MSb4rrkvInFtKY1g5ZcNoWepKTMpYQMr3+mmR4DQnGdbz/bjpRw3qRill8+haQUoyZIbGIbAptRTX7sQAhBeyCOTP/IG6u8FLud3HlVnWUwokmdWHp/35dUP3oPBkf6Uuzu7rP6H7cE4sRShtWsfDT9sG2OTrZb0ZH+Xg2p/P89CcMqBtvwiXqlcyUEFR6NuC6JpXQiWckIpquoNRAjYUCRa/T5Gm1DYDOqyXfleByCxqp0z9gqD8uzupF5nUor5qGXDmQpmCp9+G88204opqOhAmeaUO6mh/7YaYnT2buD04dst2JTlUoEcQgla6gb4I/piECcG55tS4vPQVOVl/ZQHEP270wGEIkbrH2pc1ReJ7YhsBm1GIbByi37ePtQHyCYPamYpZdNQQgNBDg0YSnMmsYCoC2YkQM4d2IxsZRBKKarSQBwanDehBJuu2giX/tFC0kDeuJ24dnpRCG3YpPPA34ld64BSV21pQToScK2w7mSM5UeQaOviLZgnEhcJ2lAazAxKiqJ87ENgc2oREpV7v/6gT7zHlr8Sh64JRDDH1UrumhS58GX9lvG4ryJRTT5vMjuKNMqvSy5ZCLfeqYtZ4WXMpSxSFgFZ1kND2xGHUcrGsuesE3D4NZUc5qbnm0e8DWdAqZXF7HwYqXmue7l/bQGE6MuSGxiGwKbUUlcl9bKHpSaaJPPS7lbs6qQGyrdrNnayZuHo+mjJK2BOLVlLsIxg7cP9rFy635r1ZeNYajAYIlLIxTXKXGJUfkDH+scT9FYfhGYYRiqQFFkt57PpcyjYRiw4Dft1useqwPeSMY2BDajEo9DMN3npdmvhOhuv3SK1QjElAZ48KX9vHW4z3qOSxMkU4alFAnQ7o9S5tboSRiUuQUaEExINE0QSYGmqalA04RdeDYKKSQeZ8aKzJqBdS8fsOJAiy+dnO5XnLs4OG9CEbdfPJGHXjnEnmCcBp+XtvTO03zd0WgATGxDYDMqOZqeixAikz2EcuqcM96DJhxsO9SX8zqRJJS5VargsksnsuqPB9kTjDM9vbuY7vPSktV2sBDZK0qbkYXHIVTbUilprHRbzWWy04TNhjK6rtPcHe1nBMrdgjsunUyx28ndV03NMiD7ESJBY6WbdS8fOOquY6RjGwKbUcvRGnxYWSF+qK/ysOjiSSz4dXu/43TUDqA9EOXaZzsIx3XKPRp3XDKJhIElcJdtbPIn/vu2dtLsj9Lk87L+szWn6uPanCBWkoA5KYtMrUl2mjAon38wLvnu7/f1e51oSvXBnu7zsuSyKdak31jlYeMnG5BSMv/XbQWlKkYLtiGwOS0ZOCtEEojqlpZ8mQt6kxCKSyTqhxyOG6zcup+94US/FV6+z/nWiybxxoFekoa0eiPbDD/m99TsjxKO6SrX3x8jljJY+/J+grHMqt8BlLi1nJ1AhUcAAoGkNwn+qE5LIG4lI5iuJoD1rxykJ27g0rB2HaONQRmCSCTC+vXrOXLkCOPGjePWW2+ltLQ055j29nYef/xxotEomqbxmc98hosuugiAjRs3smPHDoqLiwGYP38+9fX1gxmSjY1F/o5hyWVT+N4L++iOZtxDfancBiMCKPdo7AnG8cf6r/AKCZVlAoqjR1vmdMXcrRlG5ntSU7oqGFP3x3K+qbPHudnelch5nTX/q44nXjtMazBOiQscmoOmKk9OMoL5v5KqkPiKHCy8ZMqocwvBIA3BM888wznnnMM111zDM888wzPPPMOXv/zlnGPcbjcLFixg0qRJ+P1+Fi9ezHnnnUdJSQkAX/nKV7jwwgsHMwwbm+MiYcCecDznvmxdMA04Z0IRKy6fzOqXDiLyYgOFhMoqPA5mTyy24ghFLge5UQibocLarfljpAyD3nQ2mPkVOwQ89NIBInFlHJwanDOhmJsunMg3ftma81pLN+8lkjBIGuArcrDu4/VW/+F+O80swzCapKezGZQhePXVV7n77rsBuPzyy7n77rv7GYLJkydbf/t8PioqKgiHw5YhsLEZKsxMo93dUXRDEs4TojOAHUei3PfiAdpDceorTfExNckEYyla/LH05KCx8OLJaJrGsiumjqomJKcrVoZQNKU0hPI2aFMrPLQH41ZV+cxqD8svn2LJR2S7hgJp15ED5e7JbkI/UP3BaP7+B2UIQqEQVVVVAFRWVhIKhY56fHNzM6lUigkTJlj3/cd//Ac///nPOfvss/mXf/kXXC5Xwedu3ryZzZs3A7BmzRpqak59UM7pdA7J+5wK7LEXZt1na4ilDNwaLHp2J3/dE8w9QAjagjECMYNgtI/5v25j5vhSNKHx6t4gSUPidgjOnFhO7cTxAESTOiVAkcuB0+mkurqaWMoYdeqlo+2akVJaMRmHw8Hk8TXMGH+E3UciGBKQBoGoqhh3Crj742ey5v+18JeOIBLY0RXnwT938eA/zOL/+3oln3j8r6SMTOmgVWnuclNSUWWlJ59sRsJ5P6YhWLlyJcFgsN/9X/ziF3Numyl7AxEIBHj00UeZP38+mqa2T1/60peorKwklUrxwx/+kF/96ld87nOfK/j8uXPnMnfuXOt2V1fXsYY+aGpqaobkfU4F9tiPTi+w+KJxfP1giHBC4kDVCkhD0hNXS0kDFST8U0eIchck0nHgErfGtbOr2HvgEOtePsC2g72YEhfrPnMeC3/x1qhMJRyua2agyt9jPSc7BfRDdZUkk0la/DEaqrwsu2wyCQMe2LqHlkAchMa3/uNNGn0ezhnv4e3DSjPovUM9dB46gluDUreDUExljU2rcFv1Jn/ZE+ILP36V6T7vKfk+h/K8Z3tosjmmIVixYsWAj1VUVBAIBKiqqiIQCFBeXl7wuL6+PtasWcM///M/M2PGDOt+czfhcrm48soree655441HBubk0bCUKmBoETmSlwQjBcWk8hufWkYkuueyfiU1UsoiYtgLFWwgMmmMMdT+VuIuC5p9ketFNCdB8NEEpKkIQnHei1tqZZgklACBAYSA/wwrcKNcuULGn0qBhTXJZpQuwCHJrj14slc/6tWkoa6z8waOl2/z0FFNubMmcPWrVsB2Lp1Kx/60If6HZNKpXjooYe47LLL+gWFA4EAoC6GV199lalTpw5mODY274N0IBiYVuml0n303B8BGLpBKqtTlUtTVcsNVV4qPCq7pLrYOWp1Z4aS7Mrf4+3qJaVESklTldcy2rohSaWNQkrCjc+18ZWfZyqEJVDpdVBf5aEtqOI8JS64LSsGZH5v031eKr1OZk8spsqrUenRqC46vb/PQcUIrrnmGtavX88LL7xgpY8CtLS08Pvf/54bbriBV155hZ07d9LT08OWLVuATJroI488QjgcBmDatGlcf/31g/s0NjYngNepMXtSMc3dUSSCveEEtZVFBC1tIpg1zsN7XXFr0pfk9kI+b4KXhZdM4aGXDtAWiLH0v3cVLEI7Fu/HPTJayf6sHofIybo51kSbnRk0tcJNpUcQiMt0k6EM4biRc1sDhDSQho5ZRxaMS9Zs7cTlELQGVXLAhk/UW7EAMwnArTGqdYSOByFlfmx9dLB///5T/h62n314ONVjz68MDsZSXPdMK0lDKj2irIrTKq9GIGbgTLuOepOZnUCl18H3P9WAEIJ5z7Xij+qMK3Wz4RP1eJ3acU/u79c9crIZimum0GcFjtsIxlIG855tVZlBqJoPDQhkZfxUeTUMiZU++oEaL7u61S7Ale5MVmjjIQRcMKmEFVcM7fkfCTGC0Zn0amPzPjEnonnPtXLfViUnoHK/CxeFRRIGLk1w/uQSfvDp6cyeVIxLtTugN6Gz9qVOq7bAKaCpuli1QTQMVv5hHzc+28J9W/dxtPXW+3GPjFYKfVaVjlk4wypbHA7Uqry+ypNuMal6Tq/+aB1VRU6EgAqPxkP/u46GCjelLsFZ44tYfOkkzHYUSaOwEVDvBe3prnTZ7zkWsCUmbMYUhdUoNasorLHKw+7uGKG4gQCrZuC2iyYhhGDhxZOJJnVu++92gglJayABAmu12dzVy30vdpLSDd44qFxMu7ujhOK6lYuev1M4UffIaOZEPutAO6Xll0/h3i37aA3EkAiWbd7HmRNKue58Hxv+dIAbnmu3agWCh6Ks+eN+NCBb/KPMCVFdGQVHuuZA0wQNVaNfQO79YBsCmzFFoYko2x9sqlOG4job/3KQ1kDcUpfcdrAPkBQ5NasYLWUYzBxXDP4YPXGDrt4khmFYEhSgAplmD9wll01h9Yv7+000p0NR0vFwIp8132hn949efkUtR/pSLPldB93RFLu7+tC0GtqCiaw2pIrtR+KUODOxHQH86ycbKHdr3L1lP++kY0LlbsGCCyex8PmOMZf1ZRsCmzHFQBNRdrWoEIKqIo1l6eMMw+Cr/9VixQ6SWY1sNIHVpWrty/tpCyaoLXPhz9IzcmiC7mgK/HCkT1Und0dzJ5qBlFRPxyBy/mct9BnNzKDGtNFurPKw9qX9tAaVYUYIK8jvK3IwY1wJZS6BnnbnCKDULawgcrbygwQefuUADiHYnjYCAmjMyvqC0393lo1tCGzGHEeTr4b+Hav6EkZOADnnWIQ1gRmGgW4YaNLIkSwwJFQXOdCl5I7/aUcijisdcaQEkY/FYIxV9mdsrPIw/yMTKXdrrPljWuq50s3GTzYAMP/XbXT3pVTRX9Z3UuFSMZm4Lq0AsQb84FMNrPvTIVr9UWrLXASPZHSm3jocw0GmCensSSUsv2IKmqaNmd1ZNrYhsLHJYqCsFqfIZAtl49Q0Egboeoq/HVCry0BU54l/mMbtv9tLIGbg0ASrPzqNxb/rwB81+omYDcRA3bVGEoM1Vtlqrv6+FF//RQsVHgeakJbeT34cZVqFO92DWhFKwp87Qlw7u8rqLqlp6nmGbhBJGOzsilvdp02EgCq3RqPPaxkB8/1G2nk+1diGwMYmC7Ni1R/VkYYSmtv454MIoX4spW4HmqaqizVNNb/xOASHormO6ZQUnFFdZKmSjksXKpm3y92aVf06UMZMfjzDramJcyStVOO6zLi65IkZq3w1V3OFH47rnDehCE1L5sRxspvL37d1H+919RFKq0dL4F//dJDzJhTRGlRdwx56+QBvHMrUhLg0KPU40A1Jb9xACGio8rD8ilrLCIxVbENgY5OFO51nLoBwQueW37TSm1RZQVVejXV/P82SGjbTHgHK3Jq1a3AKGF/i6udiWHLZFMIJgzKXYNWWfengM5w/uYRlBVbS+ZNfdpD5/RStnQrcGuhSIoT6330C82lcl7Sm1UArPWBI5dMv9zhYccUUknLgOMKyK6YSTerc8GwbobgKzHcEYzzwsQbKPQ4AvvpfzdbzXRqcN7GYBRdOwq2ppvP+qE5HOGn3osY2BDY2OSQMcAihlCcNVX3q0pQRkAgWPd+BLlX++vTqIpams4Ca/VHOm1jEd//XB9D7wmialuNikFJaE/m0chdvH45arqbm7ugxV9I5biIJ927ppD0UP6Y7Jr947mT7vhOGCphLqf7PnlSPFTvwOAQNlW6CfSl64pJzxrvRcbAvFGfNH/eDELT4YzT5vEhDWg3ml10x1TLC6/++jodfOcDecIKehMENz7ZR4XWw6ZP1kHYGOQX88B8a+P5fD7Pw+Q4aqzw0VXkQIjGmAsJHwzYENjZZZPc61qUSImuq8jD/wkks/G2Hle0DsO1AH8FYympVGYpGWbuljeYjPVZ/W3PVnj2RG4YBWR7rhipvv8nILKTKzpRprPKAhLpKD62BqOVDH8iI5AdikaqbVpPPW3AHkv/c4zEaZo8H0+Vluq+sHYw/Rn2V0v0v5H4xZCa/f9vhTJewFn+MnrSIXM+BPqSUpCT0xNU5L3drfPOZVlWf4db44RfO44s/eQMJhOI6CQNmTyq2DEmx20lrMG7FWzZ+ssGKPYwUN9twYhsCG5ss8t0x5kQOquexTPc8VlN4bjWyDry2N4ghobk7lrNqX3LZFBor3RiGwbQKN5qm0R6M01DlZfnlU4DMBBrXJetePpDTb1dKycZPNliaRlJCpUcwrdw1oDsm2/hIKa3X6jnQRyxlUORyFHyelJJVW/ZZbihzBX4858vc9dRXuGkPxumO6vhjKe7dso8VV+a+Tixl0OaP9n9NoK7Sy/bDZkA4E+JNGfCdX7dQV1lEMN1KNJjOzqrwOqzCvQqPw0r/HaiTmG0AMtiGwMYmj2yXTrbveOnltcRSBg/9cR+tQeVWqPQ6OW9CEc3+GL1JSTLd2GRqhdtatRuGkQ4MCyIJgzcPxXAIOGe8l0UXT0JKyco/7KPF34dE4BCCSFLmpKyaf7aHVC9lZ9od8/bhKKu27GXRpbX9JrfsYHN9pYc3D/SmH+mf/iSlpC+RIprUkVKy7WAvSUOtwI9mNLLPVyyVaewuDUlthQd/tA8poS2YK+EspTJ2kWT/1ztvQhF3XjGF1S920uxXRsXpcNAciBGI6oQS6nObOIB5//k29eVOvnPxNCq9Tmv3cbp1EjtV2IbAxuY4EUJQ5HKw/Mo6okmduK6KnjRNpYGWuDRCcbVbaPXHLK2aYMzgoZcO0BqIWlWvKQlvH45x47MtSERWm0QJSDTyUlal0r5prHQTjqWs19EN2HYwyo3PttBY5c3JgMlfra/astdy4WT31jUMg5Vb9vHO4feQEs4aX5TV5vH43Uceh1C7Ht0gZRh0BPoo92g4hDJIZpGYEIJoUmd3up9Adlrn7AnF3HnlFBwOB0svr+XerZ20B+I0Vmk89NGpfOtX7TljcKB2Yt3RFN3RFBv+cogVV9QO+P2NtbTQ42WMx8ptbE4cKSU3PtfGN37Rwjd+2UKLP+2vFxkp5J6ktCQNJNDi76O+wo0r6xeXkioYHY4b+W9BiStXHC0Yl9zwqxZu+buJlHlyV+eGVD12Xz/Yx71bOy0DlD9RmwZLZPnqpZTcu6WT1w/0kdDVLuTNg32UuDV8RQ5mTyou2JA9X7zPnOR1CZGkQTghCSVUAHnt30/DkIIbf9XC3S/sJZFI8O1ftRKIqibyH5xczI+uaeCDk0rYG06w5o/7MQyDcMKgPRCnO5qitYB0BOTqBwG0BWKWaF++YJ3NwNg7AhubEyQU1y3/dChuMHtiMYgE9RUe/ma5X/KfI3FoGmePK6LVH6U3RU6Pg2wcAnrSLpPs1XI4Cau37iMcy0x/VV6NugoPbx2OWuqZavKnX8VuS7o+QvgzbppoUqfF35fz/hKIxA2e/EwTlV5njhvFNC5S9tcBevCP+3n9QOa1BFBf4eaRPx/kjfT9gYN9fPE/W60JXAJJXVLkctAWjCmNJr/KimoLxtJV2U4aqzw89teD/c6rU0CxSxDV1Ys1+byWXtRoqMoeKdiGwMbmOMhvpmJO0AL47iWT0DTVf+ChVw7xt86e/s9HZeyAqoQFNYkVuTR6ErlL3Ww1i7PGeXgnSxphZ1ciZxU8rdLL7RdP5MGXD7I3FLcK3LIDxeGYqoeIJJUbxsz3NwyDG59tIxjvv2LWtP6FboZhWBN0Y6Wb+koPSEl9pXL7tAVj1rEuTXDuhGKklJYRMMlfxe8Jxnjo5QOE07GPugqXZRTMKmy3lqkLUOmg9RS7nZZLqqSiiv2Hu6xq7ex4BYzMquyRhG0IbGyOQf7qcsllU/jgpEzVcJHLYWXLzBhXQrlHoyduUO7RaKrypgObkiafF103CMZiSKDMoxFJ9Pd3OIQyBpoApyZyYgXZk2i5W6BJg6/9ohUp4dyJRdx60aQcwTYpJT1xnWDGllj5/rGUQSiRPy2rbKTpPq/12UEd+8AfO3njoAr+9sSjFLsEIGjzx1j38n6afF5IB3dvv3SK1bDnWI6Zukovbf6Y9RkFKtYAqkLY41A6TqaHRwhBsdtJsdtpjfGe3+1m58GwtfofS9LeJwPbENjYHIN8zZ+EAcuvrLN2CNmP7+7q418/Xo8QImd1CiqL576tnVbj9IYqb45mjkmpO905LS7pCCWYNc7LW4djOcecPb6IfcEYbxyKWRPtmwej6YlXZR41VLqYVu7mrUNRHECpR8OhCZp8XrUj0KDcnRHHMzOZvnvpFB7+0yHm/7qNxko3Qmg0+6P0xHVrMk4Zyt1lOq5EMJGTm29KeTdUukklo4RTFMQp1I7qgZcO4I+pTKC3D/dx9rgipJQ0BxLMe7YVIx1kFmkjt+7l/VZaa1yXvHuox0qTNXtM2FlCx49tCGxsjsFAPQxMV4PHgZUVY0hY9PwemnwZv7SZehlLGZakgq9I4/ZLJrPuZVV0VVvuxunQaA/EMVDdzwTQEzfwOB3MnuBl2yFlDMpcgs5wnGAid61toILGIp1aqut6zjG6buAQDmuH0xqM01Dppi0QswK7e8JJ1r5ykFZ/JvVVCGH1V3AK1fpxx5GMARJAfbnTciUZhsE3ftlC0ExzzTufZU4l+KZpgsaqIh7+0yE6AhlDlzTgrUNRdLA+i/U/anfUku4kVuRyZGRB0jsps67CzhI6fgZlCCKRCOvXr+fIkSNW8/rS0tJ+x33hC1+grq4OUP0577jjDgAOHz7Mww8/TE9PD42Njdx00004nbZtshlZHKuZivl4KK6z6H/2cCSShEB/v3S+QSlyqRTJlX/YR1tQuZ3Wfbyehb9tt+oRUhJag3E2frIBwzBY82In2w/H0JOFHS7lLhCaRl/SIJJ3TCQFpHQIKD+RP6oTiqas7KSUoe4TxHIm1qYqD4FonzWh7+9JUOpSwWtIB3zTriavUyOcMCz9n0KKrQ3VRTiRvHU4xluHlKupv4MqI+uhoYycQ6gmP5GEQU/cYO3L+1l2ea2SBdEEUip5EFs76MQZ1Kz7zDPPcM4553DNNdfwzDPP8Mwzz/DlL3+533Fut5sHH3yw3/0//elP+cQnPsHFF1/MY489xgsvvMBHP/rRwQzJxuaUcKzVpekKmjm+FMMIF/RLFzIosZTB24f7SBrw5qEoqVSKpK6Cug6h4giN6bz/uC7Y15MsOGmaSKGx4e+n8p3n91oSFNm4NNXkRQBSqq5q5lztdAjKXBp1lV46AioLyakpKWfzGF2CP2ZQ6RGUuYUV6N7ZFWPec61M93lZfOlkKjwOa0cghMgpjnvrUOFq4ux2kmVeB5s+1YDXqWS+zSpvwzCY/+s2/FGdVitDShz1vNscm0EZgldffZW7774bgMsvv5y77767oCEohJSS7du3853vfAeAK664gv/8z/+0DYHNqEUIwZpPnknnoSP9um1lS073z8tXOUhJA659tsO616ojyGrc3lTloSfeR8ooVB8MPQmDrz3TgVMTuDSYVePhncNxa4It9ThYdMkUPA7Byi37rHRPAXywtoJ4LE5busK5ushJfZWH1iwZCA21U+hNSs4Z50ZHoz0QpS+l3Ect/jhJKXjqH5sIxXXc6b4AUkoeeOkAbx7s6zdul6Z2FBVFym0ViBk4hEDTNDRNs1b3Xg2kzNU2Ms9zofNuc/wMyhCEQiGqqqoAqKysJBQKFTwumUyyePFiHA4Hn/70p/nwhz9MT08PxcXFOBzKf+rz+fD7/QO+1+bNm9m8eTMAa9asoaamZjBDPy6cTueQvM+pwB778OB0OqmdON66LaVk8XM7+eueAAAfqqvk/k/NyjESH57WxbuHI4SiSVJZi3iJ8vm3h1OUVFQhhGDdZ6uJpQz6Eim+8e9v5PRGzn5e0lCCeS63B02Lo1uvK5g0rppDkQTtoYzImwa8dyhMIKZW/y4H/Nc3zqeqyMUdz+3gT+1BJFBe5EQg8Ud19vZK/s+Xz+XO3+7iLx1BAAyhXj+uS9Y+/x67jkSYMa6Euz82k0c+N547fv0u7x3uUbpHhqTC6+CsSeW8d6SXGeNKAHjvSC8zx5cyZcK4gpP6us/WWG4o8/H88z6aGAnX+zENwcqVKwkGg/3u/+IXv5hzW4iBLfGmTZvw+XwcOnSIe+65h7q6OoqLi09ooHPnzmXu3LnW7a6urhN6/vuhpqZmSN7nVGCPfXjIH3ssZbDjYIhEenm/82CYzkNHcnYFiy4cx8o/xHmzt7/wjksTTCt3suzZd3KKo6SU6EamP+/Z4z28fTie81xDQktXhLPGF1u9Dww9xT888arlw1fvoTYd/liuFYoE/QQCggVzqnn3UA/+qI6GqhsQIkFDhQu/3897h3syrqNkkpv+cxsdwVi6j4Mk2Jfkn576K2ekZbtDcR+3/KaVYFwFqG/6UA1CjOsnld3d3Z3zefIrpbNL906na+ZUMnny5IL3H9MQrFixYsDHKioqCAQCVFVVEQgEKC8vL3icz+cDYMKECcyaNYv29nY+8pGP0NfXh67rOBwO/H6/dZyNzemCkrX20nOgD5AFfdgJAzrCqlCswgWTy93s7Far9VK3YMGFk1j4fEdOcRSoSRSg3K2CpGaRmxnKEAJLDnvV1k7agnHqyt28meej/8Gn6vnBq4dpCyVJpfR0No+XG55toyeh6iFm1BQh/HHqqzwsu2wysZRBOK6j6zqNVR5C0T50VEV09utXeR0EYjrBmOS1zl6iSZ1NfzlEb1IZoKllSj3V4cgYxmNJatuVwiefQbmG5syZw9atW7nmmmvYunUrH/rQh/odE4lE8Hg8uFwuwuEwu3bt4tOf/jRCCM466yz+/Oc/c/HFF7NlyxbmzJkzmOHY2Iw4hBAsS6uWQuG2lKZYWziWok8XlHpcXDDRQXsowXSflwqPo1/6qpQSiVA9eoXGnlASiSoGE0IQiBn4vA4WXqIE3JZfUUs4YVDqhG8805qjb6RpGgsvmUJ1dTWRoJ+4Llm9ZR/hdCA4HDeY9+EJPPqng7R2R7nnD3t553CmAKzCo1Fe5CAS13P0gBzA3VdO4Tu/3QMoI9WTUBW/ptjcm4djfPOZVp76x6aC/QoGkrSwK4VPLoMyBNdccw3r16/nhRdesNJHAVpaWvj973/PDTfcQGdnJ4899hiapmEYBtdccw21tUod8F/+5V94+OGH+b//9//S0NDAVVddNfhPZGMzwsiuJcgm29Wx8JIpzHuuVWXDpNNFsxun5GcbxXWpOqmlM3umVbghpCpxhabRmqUymt0dranKw5P/UG/p/TgF/OvL+9nTk2TWpG4WfqRGtZDMChCXe1Su/luH+khJCBzKLW4LpzOPXJrAKZTwnFNTctJTKzyUuwXhhKTcLRhX7KSpyoMhVV8HUNpN4YRBhUfkfMb8xjqq2tiuFD4VDMoQlJWVceedd/a7v6mpiaamJgBmzpzJ2rVrCz5/woQJrF69ejBDsLEZlRSSrcjOhsnfOeSnr1qd1AJY3cdA1RAszetnnK+706sLKoocqmZAwLZ01fLOg2Hu3RqntbuPnqxK4Mc/Vcf9rxwqWBMApI2A6gh220WTssarVvgzxxXT7I8x3edF01TFr+o33Jo2AA7KXKKf6yeuS5rTQnlSSjZ9qtHuKnaKsKu3bGyGgUKyFSciiZC9S5BSWrn1+GPW5Gq+RnYhW2OVh0df2W+txk1XjgDqfSW0d0cI5InQHYpK2gO5gWgNWD13Cks3d6Kjgs23XTTJ0v8xiaUMWgNxAll5/16nRrHbyY8/M51wwqDcrWoF8l0/+RXDHoco6D6yGTy2IbCxGQaOJVtxPJjHSylzJC5u++/2HImLfKNRSAiuzKPR7u9Fl1DlEZYxcGowudSZ7uMsSRkSgeSMmmLqqorQNIFuyAGzBo8m/qZpGpXpIgGPkP2Oy3Z/2RXDpxbbENjYDAPHkq14P68Viuvc9t/tdEdT/SQu8o2GWZTm1GDWuCL2hFRjd1+Rg/WfaKLUCd0xg5oih9UtrFAf5+wG8YUa2Bzv5yx0nMeB5f6y4wKnFtsQ2NgMEydTFM2UuDjWxCmEYNkVU63OXabPffWLnbSFkjRUuCy30oTS3OcV6uO87Dgn+eP5nPnHnUxjaXN0bENgYzMCyS+eOh5OZPWdn8W09PJaSit9RIL+E5pwT7XCp60gOjTYhsDGZoQxmOKp9ztxmsah1151j0lsQ2BjM8LIzyiyi6dsTjW2IbCxGWEMd5vF9+OWshnd2IbAxmaEMZxBUlvTZ2xiGwIbmxHIcAVJbbfU2MQ2BDY2NhbD7ZayGR5sQ2BjY2Nh5+6PTWxDYGNjk4Oduz/2sJU7bGxsbMY4tiGwsbGxGePYhsDGxsZmjGMbAhsbG5sxjm0IbGxsbMY4tiGwsbGxGeMIKeUAnUhtbGxsbMYC9o7gKCxevHi4h/C+scc+PNhjHx7ssQ8O2xDY2NjYjHFsQ2BjY2MzxrENwVGYO3fucA/hfWOPfXiwxz482GMfHHaw2MbGxmaMY+8IbGxsbMY4tiGwsbGxGePYMtRZ/OlPf+I///M/6ezs5L777qOpqangcdu2beOpp57CMAyuvvpqrrnmmqEdaAEikQjr16/nyJEjjBs3jltvvZXS0tJ+x33hC1+grq4OgJqaGu64446hHqrFsc5jMplkw4YNtLa2UlZWxi233ML48eOHZ7B5HGvsW7Zs4emnn8bn8wHwsY99jKuvvnoYRprLpk2beP3116moqGDt2rX9HpdS8tRTT/HGG2/g8XiYN28ejY2NwzDS/hxr7Nu3b+eBBx6wrpGPfOQjfO5znxvqYfajq6uLjRs3EgwGEUIwd+5cPv7xj+ccM+znXdpY7N27V3Z2dsq77rpLNjc3FzxG13W5YMECefDgQZlMJuWiRYvk3r17h3ik/Xn66aflL3/5SymllL/85S/l008/XfC4L3/5y0M4qoE5nvP4/PPPyx/+8IdSSilfeukluW7duuEYaj+OZ+x/+MMf5BNPPDFMIxyY7du3y5aWFnnbbbcVfPxvf/ubXLVqlTQMQ+7atUsuWbJkiEc4MMca+zvvvCNXr149xKM6Nn6/X7a0tEgppezr65M333xzv+tluM+77RrKora2lsmTJx/1mObmZiZOnMiECRNwOp1cdNFFvPrqq0M0woF59dVXufzyywG4/PLLR8SYjsbxnMfXXnuNK664AoALL7yQd955BzkCchtG6jVwPMyaNavgTtHktdde47LLLkMIwYwZM+jt7SUQCAzhCAfmWGMfqVRVVVmr+6KiIqZMmYLf7885ZrjPu+0aOkH8fj/V1dXW7erqanbv3j2MI1KEQiGqqqoAqKysJBQKFTwumUyyePFiHA4Hn/70p/nwhz88lMO0OJ7zmH2Mw+GguLiYnp4eysvLh3Ss+RzvNfCXv/yFnTt3MmnSJL72ta9RU1MzlMN8X/j9/pxxVldX4/f7rWtrpPPee+9x++23U1VVxVe+8hWmTp063EPK4fDhw7S1tTF9+vSc+4f7vI85Q7By5UqCwWC/+7/4xS/yoQ99aOgHdAIcbezZCDFwr9lNmzbh8/k4dOgQ99xzD3V1dUycOPFUDHdM88EPfpCLL74Yl8vF73//ezZu3Mhdd9013MM6rWloaGDTpk14vV5ef/11HnzwQR555JHhHpZFLBZj7dq1fP3rX6e4uHi4h5PDmDMEK1asGNTzfT4f3d3d1u3u7m4rIHiqOdrYKyoqCAQCVFVVEQgEBlw1m2OdMGECs2bNor29fVgMwfGcR/OY6upqdF2nr6+PsrKyoR5qP45n7NnjvPrqq/npT386ZOMbDD6fj66uLuv2UF7fgyV7cr3gggt48sknCYfDw76DBEilUqxdu5ZLL72Uj3zkI/0eH+7zbscITpCmpiYOHDjA4cOHSaVSvPLKK8yZM2e4h8WcOXPYunUrAFu3bi24u4lEIiSTSQDC4TC7du2itrZ2SMdpcjzn8YMf/CBbtmwB4M9//jNnnXXWgDudoeR4xp7t333ttdeG7TyfKHPmzOHFF19ESsl7771HcXHxqHELBYNBK4bU3NyMYRgjYuEgpeQHP/gBU6ZM4ZOf/GTBY4b7vNuVxVn89a9/5Uc/+hHhcJiSkhLq6+tZtmwZfr+fH/7whyxZsgSA119/nX/7t3/DMAyuvPJKPvOZzwzzyKGnp4f169fT1dWVkz7a0tLC73//e2644QZ27drFY489hqZpGIbBJz7xCa666qphG3Oh8/izn/2MpqYm5syZQyKRYMOGDbS1tVFaWsott9zChAkThm282Rxr7P/+7//Oa6+9hsPhoLS0lGuvvZYpU6YM97B5+OGH2bFjBz09PVRUVPD5z3+eVCoFwEc/+lGklDz55JO8+eabuN1u5s2bN2Aa9VBzrLE///zz/O53v8PhcOB2u/nqV7/KzJkzh3nU8O6773LnnXdSV1dnLWT++Z//2doBjITzbhsCGxsbmzGO7RqysbGxGePYhsDGxsZmjGMbAhsbG5sxjm0IbGxsbMY4tiGwsbGxGePYhsDGxsZmjGMbAhsbG5sxzv8PjLOzPA/hXhwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 生成月牙型非凸集\n",
    "from sklearn import datasets\n",
    "x, y = datasets.make_moons(n_samples=2000, shuffle=True,\n",
    "                  noise=0.05, random_state=None)\n",
    "for _,c in enumerate(np.unique(y)):\n",
    "    plt.scatter(x[y==c,0],x[y==c,1],s=7)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXkAAAD4CAYAAAAJmJb0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOy9eXQc13mn/dyq3gAC6MbCDeACgLRleZFkxfKihbLGzIytyImdnO+Mx1nG8SQea4kzkqxY1OKJrY2OFs/ElpOZ2HEyyfnOdzKTONbqjGlLokh50UZtlhmRBLiCAIHesPRWde/3x+2qXtANNEg0CRL3OUdHBFBdVV1o/O57f+973yuUUgqDwWAwnJNYZ/oGDAaDwdA8jMgbDAbDOYwReYPBYDiHMSJvMBgM5zBG5A0Gg+Ecxoi8wWAwnMMEzvQNVHPs2LFFO1dPTw/j4+OLdr5mY+63uZxt9wtn3z2b+20u9e63t7e37mtMJG8wGAznMEbkDQaD4RzGiLzBYDCcwxiRNxgMhnMYI/IGg8FwDmNE3mAwGM5hjMgbDAbDOYwReYNhHpRSZB2J6cptOBtZcouhDIalhFKKe585yv5Elk2dYW67ch1CiDN9WwZDwxiRNywZlFLkXEXYFqckpHOdZ6HXyLmK/YksEzOO/3UkYETecPZgRN6wJFisiHmu85zMNcK2YFNnGIBNnWHCthF4w9mFEXnDkmCxIua5zpN15IKvIYTgtivXLcoMo5zFmrUYDPNhRN6wJFisiHmu80QC1kldQwixqBaN8fkNpxMj8oYlwWJFzHOdp1lR+UIxPr/hdGJE3rBkWKyIea7zeD/zyiLPhNgbn99wOjEib1h2nGm7ZKnMKAzLAyPyhnOChSQyl4Jdstg+v8FQDyPyhrOehUbmxi4xLCeMyBtOmsX2tU+2rHC+yNw7r9eWwNglhuWEEXnDSaGU4tbH3uTN4+lF8bVPxSefKzIvP+/5a8b54gdXIoRoWgK21kBlauINZ5Kmi7yUkltvvZWuri5uvfXWZl/OcJrIuYq9Y1OL5mufik8+V2Reft69Y1Pk3B7/vI0OLNUiXU+0a50PMDXxhjNK00X+iSeeoK+vj0wm0+xLGU4jYVtw3qo2pEwviq9dKxpfSAQshCBsUzwe//jy8563qq3iPhsZWKqF+9YrernnmWMMJbNs7opUiHat8wFnPMlrWN40VeQnJiZ46aWX+M3f/E0ee+yxZl7KcJoRQrD9mvM5Onpi0WyImy7rBfTKVFhYBFwvis65im1b+shL6Fu9komJCf815QPAYGcYpbRvX28moJTirmeOsef4NErBvnjWF23vtYOxEFBpGy1WktfYPoaToaki/zd/8zf8zu/8zpxR/I4dO9ixYwcA27dvp6enZ9GuHwgEFvV8zeZsvN91a1ad8nk8f3/v2BTnrWpj+zXnkym4HEgeYGLGwbIs2mJdtARt//isI/3BwOv1PpQaYmLGQQhBa0eMr/7ftyrOGQwGZz3fB3+zm0SmwNd27OOPnjjI5p4V/NnHz8eyLP9a568ZZ+/YFJu6W9k3rgXeEnDe6g76Vq8E8O//7StX8P/9xwtoCdq+ED/0Wz3+/S5UnL3PRK1ntBSF/mz8DJ/r99s0kX/xxReJRqMMDg7yxhtv1D1u69atbN261f96fHx80e6hp6dnUc/XbJbr/WYdyZvH00zMOEiZ5sjxMR7aPUIq4xC0oL8jwGRiggkJIQvu23mM/Yksg51hUNoO6Y+G2NgRIDmTJ5Vx2PbI6wwlssQzLlKmOTp6gr7VK/2Zh3fdB57V0f9UXlGQivHpPP/l/+zhzqvW+yL6xQ+uJOt088CzR0llCgQtwQVrWvmTD+mZQeX9S+LxuD8AlTN9Es/Ge8bVz+jo6Ima1zjTLNfP8Omi3v329vbWfU3TRH7v3r288MILvPzyy+TzeTKZDH/+53/OF77whWZd0rAEacRiqPbjQQt3QSq6WmxuuqzXF/b+WNgXb88iSWQliWyGC1a10B62iWdchhM5BmJhhMizqTNMyMKvBiofHOIZt+p+YSiZq/DOvWqcoVSegoSuFotbLu/1o/16+YSsIwFOKoKf7xmZ2n5DozRN5D/96U/z6U9/GoA33niDRx991Aj8MqPR6hUhBNu29JHOSzpCWhB9rzwWIucq9sUzWpAVvngPxEIUHEkim0UpOJjIsD7WAirPpq6w78WHbVFRDVQ+OJTTHhIELMHmrsgsES0X2YFYyC+7tCxrVnUPwD1PH2HP8WlAcNHaVm5vIKcw12BoavsNJ4upkzc0jUbLIqWU3P3MUYYTWfo7I9xxZR+3XbmOrCN5aPcINz95EKmgq8X2xTvrSB7YPcKRyQIdIYuZgiSZh+RYhmjY4tYrdKQdKToaYVtX10g3jasUUzlJ0IJCmc4HbYuvXz1ANGzPElFPZDMFl2sfGeL3v7efaNjmu5/c5At9dd96fW7F/niWrCOLFUC1d6tqdDA0lTmGhXJaRP5d73oX73rXu07HpQxLiJAF/dEQSqmai5RyriJkwd1PH+WlkWkUEM9Mc9dTh/nyv9mAEEIPEhmH7pYAD13dTzSsk68P7BrhpWKVS1fEYkXIIpXTip3OScYzLqtWaIX3ot/t15zP/qOj3PjEEI6CrojNho4ge0azALhS+TOJajz7JZ1zSeVcFJDKuaTzklik0hv3ov7J3AwgGOwM89DukboifrJrBBqt3zcsb0wkb1g0ykUGdIJ0OJlnoDPCti19Nbfh64+GGErkUN45gFdHM341SrkP3RGy/PYEQ0lt0QgB/Z0RBIoXRzL+Ob70g2E2d0cQwvLF9aHf6iEatovnzLGpM8x/+dAafvsfDwCQykmyjqQ1VCna3v2+PDKNUgpbgKsgGrIIWcwquxRCcPuH1/uePMD1jw35Il4d1c81GM71rMuj/21b+vy8hVl0ZSjHiLxhUagWnZsu6/WjcISOToWUvj9eXns+0BVBTsyQzJV6y0gpybmCW6/oZbKgWGErvvLjIxxM6UVIm7oiEM8yEAtzx4e1V33tI/t9nz2Rk+yLZxFC+MlVT3SFKIq4EP6CJQ8t8pXvzcsJFKQ+1gb+4uP9fPvFMa5/bIhNXZFZnrsQoqLks7we/8FdxziQzFWIc63BcC6qo/90fuFbGxqWB0bkz3GaOYUv7/tSLTrAbGFL5OjvDHPzpWsqFg1t29JHzlU8uOsYe47PAIr//P39WAIUAlsI0jmJVwezL57lL359U0U0HBGKt3W3sD+ew5WSqbxkMidZEbLobrH9CPlozmV/XFfViERu1jOpVZYYsrSV4yEEhAMWrxzXwj85MkPWkb6oV1OeNFVKVUT1vjgXB8O8hEiNysjq32N1tU1H6OS2NjSc+xiRP4dp5uYY1Q3Ktm3pmyUyN13W64vTTU8ME8+6TGQcfu8f93PR2lYevmbAF1Uh4MZL13D9Y0MksqosIaqK/5VY36EHiPKBy6vQSeVc8vk8n3vsMKAtmHf1hPjS5WvZ9vgv+cVICqmguyXAps4w0bBNNGyRzkk6wlaFyJd3r7TKHtuFa1YUjys3mU6uXLQRca73eyyvSLIsy1TfGGpiRP4cppmbY1SXJOZc5YtMUCjufuYoQ/EsErAFvtUB4Cg4UIyilVLc/fRRhhIZXCWYykksQNa5rgAEiusePcCmzjC3f3i9b+985UeHeHUsO+u1b4zn+fKPDjGWgXjGpTNi8cBHN9IStMk6kpmCQgEzBcVM3iEQCFQuuoqFWd8eJJXNYQkQQieM37myhYOJLINdEUJW/TYMtfxzr7TTi/LL/fvq3Eat32PYpqYHbywaQzVG5M9hFnsBTbn4hG3B21euIDlTYDLn8sCuo9xc7D1z97PHeHl0RidGqY7D9fcGYyFdWfNMqbKmHisEBEI62m4PW7xxIoejYDI34ydo7376CHvGsnXP8cZ4no6wRdCCqZzkjx8fwkInUL0BqCAVv/uPB4hGLAY7Iwwn80xkHOIzjn9/UsGBeI7P/vMQqZxLQMDBVJ57dh7zF2lByf/3ntn+eNGSoWjJVNlCXvXNYCw0K1lc6/e4FHa3MpwdGJE/h1nIApr5rIZalsGffvQ8/v3fPE8847JnZIbf+8d9eLJe3J+DtpDFZL4yto612Nx8uY5mhxPZOQUeYFpBhytRgOVVsigF6JlAMuvwyujMvM8jnSvdRyFf+6oSSGQlQ8kcA50RJIpE2arYANDXEeL1MV3J4yiYyLhAlg0dISDPYGeY+3ce4dWxDAWpE7VtYcu3iKoH2+pENFAxWNT6PYbtxWt8Zji3MSJ/jtPIFL6e51su/LUix56gzeauCPviWdJZ11/8E7QEnRHBYGeEbVes5T99f4hUThIQ+n6mci4P7T7GrVf0ksxWthWw0UJrFcsUPaaK+VxXKd69soXhVI7N3S08tHuEffEMbj1/5ySIBmFDNMytl63m2JTDHz95yP+ZBA7GMxUDU1eLjavgUCrPQFeEay9Zxee+r2vxAVxgOi/59icHiEX0n1z5RiUV3TBjIRACkcj54j1RPDYSsCqS3caDNzSCEXlDXc/3nqePVAh/deRY7ic/8OwRXhnN4C3jv/myXr9ny9/85mbSeUlQKG54fJh4xuVAMs/YdKFCyN/eFeLEdIFETtERtiraDnj/mizAq2MZ3rO6lRs/tIY/euIgiaweQFptSJeKexZER1CQLii6IhaOhFeOz/D//O8Ds46TVF4jIPSK3am80hbS8RlufvKgL/A+QlfpQG3v3hPskFXqQx+2Bdse/2VFcrvah1+KTcoMSwsj8sscvw96lYBnHcme49MUpPa+yxOr1VUtLUGbO67aULchl55N6IVDm7si7E/kGIyF+B/Pj1Xcy7/G8wQEBC1BMls/NHcV7Dk+w33PHGagWIo5GAvzOxd2819/dIhkfuHPIV3Qwpp3JdOF2XmEWgi0a5TMKYIWdEZsJrMuiZwiKKBQdhJHwrWPDvGXvz5Y00v3LJjqtQblye1UzjU+vGHBGJFfxpTbNIOxkF/SWBJoL21aEvR6olK++KccKaWuninupORVliilFy9V4yiIhgSp3Pwy+/qJPBf3BvnG1Rv5/CMH+OMn527m22rrdzLt1j9mqjDvZX0UpRmGlLChPcCQK0nnFS1BwTtjEYaSGdLFQSedl+RcVddLr/bmlVLF5HaeyZzkmz8dmTUYGwzzYUR+GVNt03gtdUFH4xetbWV/PMumYlfGhW54rZQupXypbCclr7JEKcXm7haeP1opzO1BHZW/PDr/dpEKGE7kmCoo0nUSqQAtAjIKZuYQ95PFu6oLftUP6HLMV4vJWS/PoGvy7bpeetgW9EeDpDIO6azLQ8+NcM/H382n/vYF4hmXoWSeh68ZqNvozGCohRH5Zcx8JZZeSaTXRKxWNC6lrFiQU07OVbp6pih8/VFdNpl1JCELrv/AGv74sf2kipGuLXQkvT9RvxTSwwYsS7AxFmJFANoCpeRsNZlGvBegzYasZLaf3gAdIfyIHbSlVD4A/OXH+1ndFvKfUa0ZkVKKffHSQLF/Qrdl8JLb/dGQ397YYGgUI/LLgHrlkfVKLD0b562JGQY6Iyil2DOaQSl4azzDXU8d4WA6z3mrxvjF8UlSObei7a53Dtd1yXv9YtCid+/OoxyIZ3Ek2EKBsBDo8kgvCZvOq5oLogTQFoSZghZOKRWvHp/hM98boiUw1xKqxhjsCvLaicb8mvL6/7YA3PuRddzw5BH/5wMdcCBdOj4csPQCKKHqRuDpvKwo89wQC9MStNm2pa/YijnHfTuPmuZjhgVhRP4cYK4a90bKI6tL87KO5KVjUzgKEiMzBL2ydGCqoHhtbIaChL1jU6Syuu1uMuuSzDq0hgL+itdXj8/4UakCXj6emdXDPRYWvGtVi1937uEdYqEF1aUU6Zc3E3AUoFTduveFILFR1Bf5d3QFGJmWpHLSz1QoIOvC/3yxcku2coEPCPjvzx3jcNphU9fs1bBewro9KIhGbFJZl46wxZ1X9pJ1tI8/nMzpxVSJ+gnX8vbN5StqDcsbI/JnOfP1p6lXHnnvM0fZH8/S3xnm9i29bH+21O/8xkvXVlzDE2qBXhUatARdLRbnrWrjjZE0qZxEADc+eRCBQipBKlfbAFdVWpzMKZJlAl+9QrY8Nj8ZG2UhvHlitk0UEZAtXveX8Uo/yLsdR8EvTmTpCFtMFgeAcqTC71lPvCTS5S2MQfGe1a185zcGmHK04G9/doSh1CH6OwLzJlz9z0E8i1vstbO5K2KifoMR+bOd+Za3110SX1xmH886fPXpoxxK5fxVlkIILlrTUqx71xSkjqQ7IjabuiLcdOlaenp6OHHiBNc/Pkwi4/pRfTXv7AkRsgQH0wVcOXeSNBaxkUoxmZOnaL4sHAW8e2WY10/kAIiGLda22fxyYn4Lx1HwrV/bSDrn8if/9zBTZfWT5e+jv0yks47krYkZv63CSyMz3LvzGHdetb7i9yqlrJtwLW+i5nWz9BYE70/kTJmlwYj82c58ydN6S+L7O8PEsw5KwaFkjoHO0qbXkYDFHVdtIFNwmck73PjkQZy8oi1k+WWW9+085keZm7si7BmZwZEKG6iO4d+KF3jP6hbu3bqGbTuO1DiiRCKrG4hFLJipUvlWuzkVMh4SfIEH3cEylWt8qPn6T0Z5aWQGhfbpv/6xDVz3+CHfnrpgVYQbPrDaL4/8s2ePMll1/gOJLKmc63entCyLgWiw5mbgFSWwnWG9WlbhR/KmzNIATRb58fFxHn74YZLJJEIItm7dytVXX93MSy47GulPU13fLoTgjiv7uPvpIwwlcxUVM+VR5kO7R3h5ZNq3SWxLV3bkJf5MwHVdHvzoRm7+wUHiGZeOFptUxq2IXgtS8dLIDNc+drCh91S9wbbHYgm8BYRF41U3jdAWEuwpCjzoSh+XyiqYN8ayfPZ7B+gICTZGw7xatIeCFryjO8SRSQeF4KYnhtnUFebWK3oJtnfiTCVr/l6rZ3FetG88eUM5TRV527b53d/9XQYHB8lkMtx6661ccMEFrFu3rpmXXXacTItZy7J8W6AU7emKmLt3jnAwmWUy51b48f2dul5eKYUjtRDHMy7f/OlI2U5NEdwOt+RBL+R90NhKUwtYERJMnmSyVbK4Ag8wVXUv7UH4y58dR5aNV94YlcorX+BBW2GWZfPgR/v44g8OMZFxkBN6c/PDkwcZiAZreuvVs7jyaL/WxiPVmD1hlwdNFfnOzk46OzsBaGlpoa+vj3g8bkR+ieANDqWkXYZUtlLYPS5a08LtW3rJubo2frqss+T+eJaBzgjprMuekWkuWNM6Z916PdrqCPeKIOQc7TOfvzLEjZf28ue7j/LK+AKWpzaBem2U39kTIhwM8NKI7oxpAe0hmC5U1s+X88pohhufPMhMQSexJ3OS10Z1FZPeCrHSW/cEuro3faM0c0MZw9LitHnyY2NjDA0NsXnz5tN1SUOD+NP+TFVHSKETirGw4JbLSxU4G6MhRJnCrY9GeG004w8Oe47PVAhZQFBRB+/RHhRMFhOUFvC1X13PdY8foprpMi1/40Se//zIcFMrbVYE9P1MzjNI1boFBeydyOOo0sqoFSHBYFeE4USW3o4wNpJXT8xusJMuK810ihtiBS3YWFxE5l9jEQTa9KNfPpwWkc9mszz44IN85jOfobW1teJnO3bsYMeOHQBs376dnp6eRbtuIBBY1PM1mzN1v0opzl8zzt7RSZIZh4JUxFoCvGNlKy8dnWTGgW+8EOdAMsfEjIMQgvetj/HW+DSOVBxJ5ypEt5b4VQs8gF0mXBK486lj898rzS2l7IxYbOxsYc/I3H1w5qL6/vTaggyOhMSJLLFIgA9siLJ/Yoa39bQiFbx1YgolLAIC/3fgcTCV58GfjbP9mvMRQpApuAyl9D6xlmXRFuuq2TfIq8Gvl7Q9f804e8emOG9VG32rVy5KJG/+5prLydyvUKq6cnlxcRyHr33ta1x44YVcc8018x5/7Nj8f+iN0tPTw/j4+PwHLhEW834X6rd6jcQOxDNs7Izw5Q9rG+C6Rw8Qz7h0tdhsiIY5nMoz2BkCBAeSOVIZB0dRs6pmPs7rCrA3fpK9gZcoumc+RGxBxlGlpHVVf/zvfnKQ1lCgYtvAwViIGz7Uy395/ADJYoO2oC0ouIru1gDf+vigv3Dt3meOsL/Yc75WJN9ItN8MT345/82dDurdb29vb93XNDWSV0rxl3/5l/T19TUk8IbFYb4/8Fp/3HkJw6kciZzESucpKO3Xb+qKMDkyo+vgMzNcuLaVmy7r5YbHh5mYcfTCqIjFYGcYqahY5Tof55rAg66SCQZshpM53tMVBmHxWtWuVQLdpC0SsMg6kgNJvUZBiAKRgMXbe1pLvWrCIfaOTTEYC/kJ8kYqqhqxY8yesMuDpor83r172blzJxs2bOCWW24B4D/8h//AxRdf3MzLLnvm+gOXUvp9UMqX2NeqtxdCcPNlvVz7yH4SWYULDCWy5FzFYEw32+qPBv0NQrKO9CP/ZuF1dGzy4teGeFvM4q1kZbnn6+P5sqg9z8ZYeNagd+HqiL/ZR60KGW8jFqUU33x+Qr+orA1CeTuKeiz2/r6Gs5emivw73vEO/uEf/qGZlzDUoN4fuFKKu58utf4tX2JfLzqMBCy92Om43phbIbj5yYMMdoX5f3/3YqZTiYpjvVLKDdEwt1+xhnt3Hef10Ywuu1wEZd7cFWR/otD0FgeNcCBZu57fs2VSWcnrNVomH0wXKhqNVe8KFRSK+3fpjcGn8pK8qxCJnL92oZGE60L29zWc25gVr+cg9f7Ac65iKKlb/wpRucTee12tMj3vXFlHcvOTB/1GWd45wzZ+0zO3uMQ+aIErbA6l8jqRG7Er9nONhvBbDDdCNAipAuyNn9myyXLmm69IdN+acgSQyLgVLQf0AibFXU8fYTiRxZGKybxCASFb0NVi+4P2QipijB1jACPy5yy1/sDDdqk3+UAszB1X9tWN8GrZOjpSD0MCBjvD/OkP9vLL0Ul//9G7nznKy8Xyyf0J3R5gU2dYd4msKq9RCGxUQ8nagADZwFKpRhdTLSYBUb/2vZr2kMV5PS0MJfRMJyj0moNUzuW/7TrKnrHZC8gu6oty4wd6/OoZY8EYFooR+WVEo1N4KSVf+fER9ngJQ9/WKfnFmYLLF//lkL9VXTLr8GpZfXx/TPvLt17Ry2f+ad+spmSTxZ7x82GhSxInC3PL6Hzd5NtDFuGgzdoVAY6k8ySyi5M3qGcbtQch41T+PGDBjR9azbWPDvHyyDSf+ad9CCH8+vhqAgIOTEzz0HMFbq+ydowFY2gUI/LLjFoRfnkyD+DuZ47ySllFSLWt4/W08Wq5ExmXP9t52O8jHLTglst7EUIvdqrVdVLRWMllRMBMA2HyfG3E+jtsjk5LLCQF93T0t9T2VTlTecX9u4/7z0P/f/ab83ruOwrGpwukMg5ZR9IStI0FY1gwRuSXOdXlljdd1stwIuevvLxwdUuFrVPdHhe0TL1ZbMdrF1/j2QsdIWuWH78QGhH4Rnit2AKhmZU/QUv4z6XWzKMgFYdSuVnf9+gI6mc5XdAzk9IQsASyzIazFiPyy5zqcktA++5xHcF7Au/ty/rQ7hGm8qqu/90esbjpsl6/Xa4Qgr/4+ABZR/LNn47y4imsJF3KdEYsNne3kC+4vFLDWw8IiEZs+jsjJDLTtVcFS8hJPSsJWvDuVS0cnnT8VsMGw8lgRH6ZU69Ou9y+8SL9jR1BhpO6WqazxWawq5V9Y5OkygpeNrYHufaRIVJ5SSysyy8PJHMMxkLcfFkvD+w6xr6JDOkFFMmczGpa0Nv1Ve/m1Agns1vs+o4Qt16+lryEB3aPcGAig6sg40ikgnevivClLesICsW//9/7a7Z5mK54k4I/uaKPlStXMpWMG//dcNKY8GCZ4yXzvvXxQb/uWvu++qORyrnsi2eYmHF4bSzDZL7YJTHrEg7a/LdrBglaWoBsAcqySBU7VCZzkj3HZ4hnXF4cyXDD48PYtsV/v2aQ2Z1WahOwFi7wNvDe1RFuuWIdjQbAobJ/n4xj/+pYls/+8wGuf2wIC8W3fn2Qzd0RClJX37wymuX+Xce46+ljFQK/IqA3GClHABesbq34PXiLo6DUk6bJHUkM5wgmkjfUpHzPUCkVsbBguqB9Za9L4lvjM7QEV3LR2la9YYZSvHmitPinPSiYKZRc5XjGRU1kyDmS83tCvD4+f6G8OgnFXRHUwvqfvj/c8GvyQJsNwoLJkyzF93aREiJPXsLBZMl/V8BQPEMqVynMNrB5ZSsH4hldVipgQyyCQHH9Y0O8Y/U4hXzBz5ls29LHfTuPzbsgSkpJOi/pCFlYlonlljNG5Jc59frclNoP6/40K4KwImT5S/YtAW9fuQKAmy5dy/WPDen+K0UNaw8JBqMhfjGRRyjFipCFKO7d+vlHhxu+v0ai+Or8QLqgI+uFMuXqC1roqp5Mnfr3VhsCtqhZNRS0YDAWoj0o/Ig9IKAjbOEqqC6WVEIPBsmcIhaGvvYQw4kMU3m9Qnjv2BRSSj9hnM7LeRdESSn5/e/tJ5VziYZtvvvJTUbolzFG5Jc59frceF69UorJnEsyB10t8NDVA3SELHKu7qty/WNDDHaGi75+jlTGxUXXwb9S3C+1q8Xmm7/Wz/ZnR3i1qlnXYuBVAi2WeSHRVT0twOymBHobws6gRa0hqC1kcfPlfRSU8L1QR0FfR4hjNaYItiXYGAsTz8yQzCmSxWcm0Mnc81a1kc/nEcWOk97er1BaEFXdcC6d1wuslNJ2WzoviTWyVZThnMSI/DKnXp8bz6vPOpIHdx3lQFJv8h0N61ptISV7T0z5g8M3f62fnKv485+M8PJI5aYhbrGs8HBq4dF1wAKnjmVzfneIt+J5nGLEvZCE6bt6Qrw5np/z+FoCD8XNTnJa4G3gnStDvFbcBEQqCApFQenqpESx58+RdJ7BrhZAty2YykksC97W3cJ1H1jDZ793oOp9C75+9QCb161hfHy8QsTrJca9mVhHyCIatv1IviNkBH45Y0R+mVMu5rV+1hK0uf3D68kUXHLFvjRe17HRRlAAACAASURBVMrzVrUhZZrBzjAP7R7hQDLHQDTIxWtX8OpoqZZeoHhg9whTeUlAwLtXtfLFy1bz4HPHGU7m2BgN8fpYtubq0QtWtZB33Jr+/d6JSpFuROAt4IJVEQ6m5xb4erx7ZRjbsnil2HjMBQ6XbSGVykl+/58PELT0blDvXd3CcCpfsVm614gM8PvDRyM2qayLbenVuZu7W4hFAmWJcOEnXMs7UGadWvaNxXc/ucl48gbAiLyhyFzdDZVSXPvo0CyPd/s153N09ARKKb/FsOu6fO3f9dMWFDy4e4ThVJ6BWLhYeqmtmy9t6cWyLL581Xpf9O7deZR/Hc/4yUsPR8HeeO0EbS2Rbg/OnTiVwFsTWeqtiWoLwFeu6uWvX57gjfFS4jQWFgx0tXLLZWu44fHhitcMxMLsOZ7xZy9p7z3Eczz88YFiAzIq9mJtKdNdIYQvyu1BQUGJWW0L6uVO6s3ELMsyFo0BMCJvYP4NJubzeENWqdtiMqf4/CNDRCM2f/2JQQpK+CKu4opNXRE/6i8Xq9u29HHXU0d46XjJs7eAV6ta9UZDMFWovZ0gQNad2+KB6nr0SqYc+OrO0eLuVyUKjmI4keHrz40w2BkmndU2zHtWhbnl8l6ue7S0m5NHf3HdAcy2VDwBL/fTYxH951irvLTe78j0szHMhxF5w7wbTNTyeJVS3PrYm7x5PE1/LIwoq9lW6MFgsqBbDCtVKlNxXcnBVEG3K6bUqjiVczmQKAl6QNRu/jXY2cK+eGaOaF3gVPf3XSCpnMue45WDy7QL065kfyLHQGeEjoiNK+FQ2uHrPznOps4wLx4v5RxsATd+cJX/HmsJ9EI25J7rd2T62Rjmwoi8Yd5o0LJme7xZR7J3TCdelVRIKitcOkI27cFSH3tvizvIM9AZAaHbFUspuWfXCHtGZirEuZbAC+CGD67mc2X1716yVQAXrAoTCgbYN5EhkW3ccX9Xd4hr37+KbTuO+INH9eU7QgLbEtp6ShS36ysepyYyfOXfrOfFJw76x7sK/uM/DXFx3wpu29JXsyImlXPZH89WDHj1xNpE7IaTxYi8AZg/Gqz2eMsTr/3REMNJ3dSsu8Wmtz3EkXSO7c8e47Yr182KQrdt6SPnKh7cdYzrHxtiMudSKGpyQOjFTN6GIuUVMwr0LklFdS0fVLwU721b+sgUXH7vH/c3vHvUGxN5bnjySE2bRAAXrmnlcCpHf2eE27f0sv3ZYxBXpLKuboOcl/zpjw/Pqu5xgf1xvV3iTZfpjZY9++aep4+wL55BKehuCdScQSmlyBRcP9ltInbDyWBE3rAgyj1kL/Fa8tyz9HdGGEpkSWRl2e5H1qwoVMjSBtbVujUQC/PKmB402kLC3yUJ4PYfHWFFyGI6L/08gPezQ2m90tS2bdrD1rzRfGkvVo0LxMI2jpRMla3UPZjU7weRo6AE27b08ZUfH2Eio/MHroR4VtLdYnP3R9bxrZ8f5xdjOSxL0B8L88CzRxlK5X1LJutI9hyfpiD1oPbtTw74lTTlz/neZ44ylBpiIBqc08oxGOai6SK/Z88evvvd7yKl5CMf+Qif+MQnmn1JQ5Oo9pAf+q0ewrbuUKnKyiW9hVH90RAha3a/eq+jpbfYynEl08XVoxetbeXaS1bxB0VLZjKvWBHQCVHQrRG6WmzaQqIi0RmgdL6QBYOdEQ4ksigEAr3S1lGVLXyl0tU0U2U9zJK5yqxs0IKBzghWUaTDtl4NfChdWrRkF/MHroI17WG++pGNfPXHh3ltLMOrx2cQQveH90og8V9Zf6pR7uNLKefd6s9gqEdTRV5KyXe+8x3uuOMOuru72bZtG+973/tYt25dMy9raBLVCcRMweXeZ47qRKhvueT45q/16xr4RI57dx4Fpfcm7Y+GsG2LA8XVm9u29JHOS256YhgXh64Wm5suXcvnHzngy197qNT/xmNDVHv5ybLWBW0RC4XF9Y8O4Sq969RgVwu3b+kl5yo+/8gBUjmJLJ5zqjg7mHJK1Tie7Nrotg0IwUVrW7mtWN/uzULCNv42ihuiYYYTGZI5hSV0mSTAwXTet4uCQkfskznJA7uOcvNlvVy4poX98ay/Mbq3xaL3nL1B0LIsBqJBs9Wf4aRpqsjv27ePNWvWsHr1agAuvfRSnn/+eSPyZynV3jrAvniGeMbFRke9kznJA7tHOJjMM5FxUHG9gCqRlcQzmYqNNfISomHb3zd2U2eYvKzccGN9e5A3Jkp18hZwKJllsCtCR8giXeyKORiLMJTMViRERSJHXurrWGUaOV1m/4A+uHxjE8sS/M/fGKAlaBMJ6J74EYuKxUie/RSy4L6dR9lfHLi8pGp/NMRkLgMILljdyoGETga/cnyG6x8bYlNXhK9fPVCxMXrWkRXrFbZt6aO9s9u0GjacEk0V+Xg8Tnd3t/91d3c3b731VsUxO3bsYMeOHQBs376dnp6eRbt+IBBY1PM1m7Phfh/6rR6yjiQSsLBt3bbREi7tkQABSzA+XeBQusDbV7ezb3yat69cgeNKfnYoBYAjFT0rgrxjdTt9q1cihPDPGbYFmYJLrOUQqUyBaCTIyEylfSLR/reddviHz/wKtz2+l+H4NOFwkPNWh9l3YopkxsGRCikE3/j5OL8cm2SqrJmYqHJKWkNWxSCAgL7VK1kRDlYI+7bHf8nesSnOW9XG9mvO94X3od9a6T8TgFsfe5PDUy6/sj7GVz92Hi1Bm22P/5Jfjk6Szjp6UEwV6Fu9kvPXJvxzdnd3M5Q6yMSMg2VZtHd2094SpiW4shm/yqZwNnyGy1kO99tUka/V77o6Itm6dStbt271vx4fH1+06/f09Czq+ZrNmbzf6iZX1V9XMw2siHYilEIqsJRiY1R74oOxEH/yoZXk5UpfuH/vSJqCVNgCHvh3G4hFAkxMTFRc/zbP748GueGjG+gIWWx/9hj/Oj7DZE757cACAja023zpkTf8zcOfP5SiPWyxMRah4Dgkc4B0eXM0XbHln0A3EfNW1rYFIFOQFIp2jQCUVNz+6OvcdOlavv7ccd9qGk7mmMi4SDfN0dETs3ZrmkZH428eT/teejyuo/CbP9BDzu3mwd3HOJDQ7R+mUwm++MGV5Fyd25hOJRiIBpFSMhANMpWM0xJcaT7DTeRcud/e3t66r2mqyHd3d1f8IU9MTNDZ2dnMSxpOguqEaqM9yyMBq8Jq2VblXXsVl5GAxYqgRSrn0hay/C6WYbs06Ff7/S1BG8sS3HRZL1lHctOTw75YtwUFb01k/Va/3p0lslInSLt0v5jBWEiH7fEsrlSkc7rRry3grz8xwH//yXEOJrJMF83zQLGlcjKn2DMyw3WPHmAqryhIhZIKV+nTucXkbvnz8wbEcktrMBaa1S7i9hq17uUJVVMLb1hsmirymzZtYmRkhLGxMbq6unjuuef4whe+0MxLGk6CaoFtpGc51F6gU6tdSs5VCKF9cCHgnp3HGK5qa1Atjq7rctezOnk72BliU1eEyZEZQLGxM+I3CAN4z+oWbAGHUgU2dZXq8AG/GsZ1XW544jDJTAGJbrx2ZLJAsmygaAkKNndHOJDIMZlzSWQVQUv32xnojOiNPZSuIErnJdGwfib3PH3EF/LbP7zefyZK6Y0/qpuHzVUlY2rhDYtNU0Xetm0++9nPcs899yCl5KqrrmL9+vXNvKThJKhOqNbqWV6P+URJKb3oaSovCVqCgc4Iw4ncrFWe3oCRKbjcv+sYv/t/9ld0a3/44wP+v0MWfPafD5DMukTDFkEBQ8VGaNu26I3HH9p9jH3xDJu6In51z3Tei+T19bySy3RxUVM6J5FS8fA1Azy0+xj7E7oM9JYr+ogELO7deZR9E3oHp5ueGGZTV5gbL13r17xP5mbIOrKYsNWWV6PP0WBoFk2vk7/44ou5+OKLm30ZwylQKyJfLNvAa2mgO1Ba3HJ5L19/bsS3eKqF74Hdlf3oBaVGX+X34bVZCFlww+PayvG23VNK8vLINAWpmByZoS0kSGQlIVvQ1aKrebwFWllHcv+uY7w0Mo1SMJzKY1n6Z3c/c5ThRI6vPzfCti19oPSgNV3QFg4Jr2WwX88z73M1GE43ZsWrAZgdkS+WbVA9S2gJ2nWFL+cqhhNZX+Bt4MI1LdxxZR+AX+UihPDbLNSKlrOOt9wJQDEQi2ClC7xjdTt/dElPxYDRErS548o+7n76CEPJHJu7Ir7FoxOturxRNy2b8fe47Wqx/VnPBWtaGUpk2dQVmZWMNfaL4UxjRH6ZMl/1zGJRL5qtJXwhS68aFUBH2OYvPt5PayiAUoq7njrCUDLL5q5IRSK4Zl4gYHHRmla/dv22K9eRl7ossrqix3vdnVetrzhH2KaG1aIHDlvAQx/rJxq2uW/nMYYSWQZiYW4rWkUGw1LCiPwyZCEtbk/lGt5iofKKm7kGF2/RkgJsS+d0AO5+5igvHdd2yr5iw6+5Zh1CCG7/8PqaCWFvNgCze7yXR+G1ttkrHzhikYCfsC63isw+HYalhhH5Zch8m4ScKv4gEs/qFgMCf/u7uUozw7Zgc1ekYvWotnByuqpF6OZljSQwvWjcK9UE/P73mzrD3HRZr/8MvLa/3v615ecofy7VA0ftaN9gWFoYkV+GzLdJyKniDyIZByFAKdifyM1bmlnLegnb6Fr8uE7A3nFlY5ZI9WzlxkvX8svRSf/a3ntXxcVcXrXMXLOaWjMGk1g1LHWMyC9Dmi1O/iCi8CP56tLMwVgIpUobg5ff22IIaflsRSm9kXg66xC09LW96ppUzuWmJ4b9BOtCZzUmsWpY6hiRX6Y0U5zKhbnak/fKFh/aPaIbdTWQEziZey2frfRHQwwncuRdRVeLzc2Xl2YD1Q3SjOViONcwIm9oCuXCXJ6M9HY4qmfbLFbVT/VAc9/Oo1gp3bZ3rgTrYsxqTlflksHQCEbkDaedejmBhVb9zCem5QPNbVeuoy3WVbNt72LOak5H5ZLBsBCMyBtOO/Wi54VU/SxUTIXQ/Wqmmyy4za5cMhgWihF5wxmhVvS8kKqfpSqmza5cMhgWihF5w5JhIf74UhXT5VJWqZTCdcEuaxdtWJoYkTcsCcr99er+L7WoTqxW96c/k5zrZZVKKZ7fNU0q4RLttLnk8hVL4rkbamNE3nDGOdlkpbdYajESnaYipnFcF1IJl2xGAS6uCwGjJEsW86sxnHFOxV9fDG/eVMQsDNuGaKcN6Ei+2GLIsEQxIm8445yKv74Y3vxSTeIuVYQQXHL5CuPJnyUYkTeccU4lWbkYic6lmsRdygghjEVzlmB+TYYlwakkK0810blcKmIMy5Omifzf/d3f8eKLLxIIBFi9ejXXXXcdK1asaNblDIZT4lyviDEsX5q2xcEFF1zAgw8+yAMPPMDatWv53ve+16xLGQyGIkopHEd39zzXOJffWzNpWiR/4YUX+v9++9vfzk9/+tNmXcpgMHDm69eVUhQKclb76MU6t6nNPzlOiyf/4x//mEsvvbTmz3bs2MGOHTsA2L59Oz09PYt23UAgsKjnazbmfpvL2Xa/sLB7LhQkk6kpshmFZSlisW6CwdOzH6FSih89cZz4+CG6esJ85Oo1iyrC5e9NCEk02kUoVLt204v4A4H58ytn22fiZO73lET+rrvuIplMzvr+pz71KS655BIA/umf/gnbtrniiitqnmPr1q1s3brV/3p8fPxUbqmCnp6eRT1fszH321zOtvuFhd2zUor2qEBKQXtUkExOnLZo13EUJ0ZnyGYUrusyOjpOYAE5jvnaJCilaOuAbAZyWckPHz/M+2tE8wuN+M+2z0S9++3t7a37mlMS+TvvvHPOnz/99NO8+OKLfPnLXzZTK4OhyZzJ+nVvgZRl6YHGWyDVSI+bRoRZCMHFH1rBU09OkssoUgkXx1EEg5XHmdW4s2na29+zZw/f//73+cpXvkI4HG7WZQwGQxlnqn7dG2BisW5/BtFoVN2oMAcCgminzXjOoZBTvPSTad5/RVvFOc1q3Nk07ePwne98B8dxuOuuuwB429vexuc+97lmXc5gMJxhhBAEg5Yvuo2Kd6PCLITg4g+28tQTaXJZSCflrHOa1bizaZrIf+Mb32jWqQ0GwxLH26S9I2YBcl7xblSYAwFBrCvgzw5qnXOu2cxybJG8zN0qg8Gw2JTbNB2dNh/+WPu8lS71hLlalOsNCCfr/S8HjMgbDIZFpdqm8cR5odTz9KsHhFPx/pcDRuQN86KUgnwOQuFlM8U916kV+S6WldGoxz7f9ebz9L3XK6VqHld9/uWalDUiv4xpRLyVUsiH74GD+2HjJqzrbzdCvwRZiEDXsy1qfe9kRL8Rj10pxc+fnfKvV10lA3MPFtWWULX3Xy+6X45JWSPyy5SGxTuf08ckJ0pfhyOn92YNc7LQBUDlEbJSDo6j2xCUR8OOo3j5pzMLbiPgrTa1bSoWQ1UPQoWC5MSoi5IwPlq75n0uUa6O8j/8sXY/WhdC4Di1o/vl2CJ5mb1dg0+j4h0Kw8ZN+t8bN+mvDUuKhS4Asm3oiFnkcy6FPLz00xkuuay1ImqGhS8q0q0NRhgbzejovDgwVA9C77uslZd/MoOSpdfqgUEipagQ9HqiXB3lVyd2l6s1Uwsj8suVBsVbCIF1/e3Gk1/CzCdotSpUylePphNaxN/7wVa/udhCzwlaqI8dzuC6ivGc40fn1YNQPg/JZCnrGQjCzn+ZBEBYglhxIKgW/PLrWpbivR9s1a+vUbmzXK2ZWhiRX2aU+/CNircQYl6LxiRnG2exa7XnErR6Vk4gIIjGLJLSpa0dXvrJNKmkREkFAmKdNu/9YKt/bPU56/npClX8P35L4OoBIxhUFA8jEAQE5LP+2Uni8vyz06SSLtGYzSVXrMCyLP+9JOIOqNKAUK8UcjlaM7Uwj2AZUdOHXwR/3SRnG+d0t8ydz8opFGDihEKpynrCE6MuT/9gqqaIlvvpJ0Zd8nmXcDhAICDoXdfC0UMZUPDyT2f8AaB8EHJd8N6yZUE0ZpNKuCgFli2IxmyScYd8DsaOuzy/a4b3X6Ffn0y4swYEfd7lt8ipUU5PH1LD0qDchz+4X3+9CKhcFob3Lfp5z0XKRTeVWJxabW/geOqJNM/vmq7YVMOLosMR6IhaWJaquA8loeYeHAq/EVj5PSqleOm5ad9PVxKeemIKKSVCCK78t2sIhfU5vbYD1dg2xLoCRFoEsS4bSwiEJejsDnDVx9q55PJWYp0lfyiVdH0Bj8YqfaNoTDdF897/z5+d8nvaN4uzbfMSI/JnEUopVC578h8uz4ePddf14cuv0cj1lFKobz8I05N67r1h0CRn58AT3UiLmLeG3HEUUspZglItMnMNHEII3ndZK9HOAKmU5IXdMyilyrpGzr62sCAY0g5dR6etZ2rF+ygUJKmkrDi+kId8Xv87GLSIdgUItwg6YqXXlg9CAJdcvoKrru7g4g+tIJV0dW4gqRdOWZbFJVesYNVafR5PyPWMoJVVa2xCYVi1xuaSy7V3773/E6MuTz0xyc+rBrvFYq4Bdali7JqzhMWwROZLolZcY8OgnlM3WmLpFCDWhfjDL5rp8hzMlxD0BPzln86QTLgIplDKJdYVqKhnT3p++OUrKjxvT5SVKp1bSkG6KKQp37IRvP+KNgoFyYs/mWH8eGlkUFL73Vf8ahuv/HyaHz2W1uezQKB/1ZXvCYSQKKUraZQrka4iMeHw1JOTRKMWqZQkV2YZebXsUko6ojYKl/aoRT7vYlmWFvrLW7U3n3B4YfcMl1yuvfn3b2nzk69SCixL0RGzkVJbPLmsYvy4U7Ms03vG3j0slLOxlfESvz2DzwLq1edKgs6ZRC2/htIJOJLxua9XVaWzGB7/uU55QrBacJ7fNU0y7lDIg5SAcEFBMu74qztPHHeQkrIKFotLLl+B4+j2u08/OemXKkopAJf2DgupXL/nu+OUBpl0WaVLMKR98VjxuBPH5az7B/1rV0Ch6Mw984Mpol0BgoE8J0bLX6NISp1ATZdd/2c7p/RxxUA4GIQTx11+9OgUwRD829/o0INTSpLL4s9QvFp321Y8v0vX8bfHLJITuhx0PqpzIh/7xMJ2WTobSzONyDeBplSa1Cl5rL5WZTS+CfGHNyPCEf9nc95X+TU2DKJVfv+c1g75HOK62xCFfMU9mEqb+akWnPd+sJVUwiWX1QnJcIugkFNIpcdcy1J1PXxdGql9cG+R0/O7ZkgmHAo5/fpQGC7+UIQXds+QjDt0RC2EJXA8cRQQ7RS877J2LEvx810zde+9kIeulZBO6n97QmyJksAHgjr6VwpSKZdYZ4D3XdaqE7dVg0ehUHnufB7CYV3Pr5RLR8xCCEk2KwiFqLBopKoU+GAIOrvtmjtTVUfijqP830UjiduzsTTznBH5pSIszao0qWW11LpWRTQ+lUbdcS2qfzPiuttQ5cfecEfNKN+6/nadSAV9nTLxrn6fyftuRb71Cx3BF9+nqbRpnGrBgTLbJWZxwSWt7N4xTWZGIiy9rV8gAD2rbX9gKBcyL8pUyqGjwyKZcMoqUShaGYJk3CGXhRNZiWUVZwwACibT+p8v7Kq0cCwLol0WiXF9sFIwMVb2ZoQW5GAgSDabRQBdPTauVEwUI3tvNpLPzfax7QC4TvFUAqQskMnYpW8A//f7kzgFLeJXXd2mtzpUClk28IXC8OGPtREMWn7VDeCLea1FVN5gW26BnUzHzKXKWXSr9VlSwtLENgCzrJZa1/KjcQVTk5CKw8H9qMkU/GKPNlR/MamFPBypOTCqbz/oP0sxhxfvHPjl7Pd5mtogLJVB/VSoJTjlUSJAz6oWTozO+NaAENpLrxVJeknW53fNkErqyhkhStUzwRBEWnRFylhRwKvlVro6oZtMlJSze6VFKil9ga+JggsvCfPmHqUrdoBUQlYIeiEPLz43jVKzz9PRCTOTglxW5xN+9Gim4ufjOdcfjAp5eOYHpaSnNziAfq97fpbx2zTomYBivDjQrFwd4H2Xt6CUVdECYbYFdnZ+pmpxToj8kuqv0oQ2AHUFrca1yqNx9VcPwKEDsL5f/4VXnVPVGhgX0O4gMHgeedeFDWXvc5He/1wivqQG9VOg3tS/vOb7I1evYXR0vKGl/uDZGDpSD4YhFNbCGY4IrvzoCpSyeN/lrbywO0Mi7tDeDvGyfaEdB/J5t+qcclaytRb5rCA+Xpo65LKVQ4hS2nevRToJXV02J7LOrJ+FQoKCU3mu6nN7FPJ6doQQ5DIKKV0KhdJAN3bc4YVdM7x/y+yGaOcq54bIL6H+Ko20AVhIFDqXoNW7lhACEWlBXn876ht3wcED8Ff3w/kXwuEDsH6Tfs3wPh3pQ0nMvWepFKzrRwVD1LtDpSiGgqU/uMVogzCviC+lQf0Uma83+sc+0VPTW66HZalS3buCaJdNOqmrbvb8LEM6qTs1/sqlEX702FSFwEOx7v3xSi8+MdHYtQMhl1hniMxMdv6Dq4hG4X2XR3j+2WnGxyoFPJ+vLeihMFWrZXUeoL1DgCUQ6H9PnJAVMxav7t577oGAqGuBVXM27izVdJF/5JFH+Pu//3u+/e1v09HR0ZRrLLX+KnNVsCw4Cp1H0Oa6lijkUYeHIJ2A1xPwrovgKw/DX38d7roRpKvnyRs3oYIhnT0LhbV//8274fAQ6pt3IW+4E6u6oDqfwx3aqweJQ6Livhppg1Dv2ZDP6f/PJeKhsJ49oCpnEecA9RKDc+GVXXoISwAKyxZc/KFWP1fy9JOT/nlzWbEoa9aCIfyk548fW7i4e9gBi5/vnCKVgp7VFhd9IMLMpOK5pzI1j+9ZZfGeSyK4jmLnv5SOsQSMjylCYcVVV+sdqZ7fratwpKufUXmFkderx7PAvOS2bauaM8jTuVp5sWiqyI+Pj/Paa6/R07OwMqWT4WSF5bSz0Cj0JGYpvlgGQ7B+oBSt/+IV+B9/BoeHYDKpDdvzLoDP34r61r0oz4f/g5v1MckJSMVR37gL9YUvV1TOqGCI4OB55F15yrMnf9GVZy9tGIS+jXqqUPfcatYs4lyglk9fD0/cX3xuivExiUAnZTs6BGmliJW93nGo6LkeadGPNZ+r9O3n44p/G+EnT2VxCvr1H9jSyrM/rF+F0yjl1Tbjo5JXns+RmKi0dgJBsAOC9g6LxIQ7a8YBpUVZ+ZweMEMhi/eXlZemkjqqf6Eo/CtXO1z4/sCsssxaIn421shDk0X+b//2b/nt3/5t7r///mZeZslTbs+oYAjW9c8jYCfXSMx7XXkJJVAqXVAK/vX10moWpWDvq/DNr8LhgzriV8Ul4ev79eCglI7oJ1Ootg7Ut+71ZyHRLz/ExMixhmwpYNZ7kFKi0kn4mz+HI0MwPaXvbSoNK9phw4Auz6w+dz6nB4Mas4ilymKX6PkVIXHX96cVMDbiEopALKbLFaG0eKojanH5r7YRDutVpb/66x3k8xAM6nt76Sczs5KlgYAeIDx+saeAbQs6YgJLKJ57ujGBFxbUyLfWZfy46w88wtJvzimAU1BI161IttbDS16Xl5fmMoq0Knn2EyeyuG4bgcD8In421shDE0X+hRdeoKuri/7+/jmP27FjBzt27ABg+/btixr1BwKB0zKLmAuv1NA58EvsgbfrbP7IIQKbzyN62/0VNoh3v+WvCQyeR2zb12Z1AaRYHTNrSpnNMH5kCJmcQHg9AV0HgkGstg7kVLr6BhEjR1DeaVIJrL96gOgdD5L43CchlYTpSdTdNxLYsAn38AFUcgLLtghIl5V962q+Z5XNkHzoT3GH9pbe94G9/vtRSjH+2V8vzTJAD0axLr8qyDpm09PRjoi0zH6mbzu/dL7evoamzWfq86D7rB9n4kSW7pURPnL1moan+fXuOZ93GR9N47qzQ/B8FiZOuMRi3QD+cScyLrt+OE3PqghbXcXjZgAAIABJREFUf22tf558zsEpSK74SDvPPT3B6EjOF+RygUfAxJiOruslPuuxpjfCyJHG7ZzymUU4LIrCq6m16KliNiKgd12EtWtXlbpjKsXK1Q4TJ7J09YQBxcSJHN2rwqxe3ePPUr1juldG/O+X87FP9OA4at6NyZvFyXyGT0nk77rrLpLJ5Kzvf+pTn+J73/sed9xxx7zn2Lp1K1u3bvW/Hh8fn+PohdHT07Oo5zsZVC6ra8mTE0jH9VeR5l3JxMixihWi3v2WvybvSsaPHfXn1ioYqoimrbL6dPI5ZCAIa9bDxDgqGdfC2dEJ/ZuRf3Az/NWD8OYefcFIC1g2qm+jPl8R+doLJD778coQKjGBk5jQEXZHJ7KvH9cOMFF8vlJKXZe/oh3+4j44uE+LtVOY9b7Hjx3V9kw6UfmwWlfocLRlBVgWct0A45NTiKnp2Yu+/vAWXcYZCjMx0Vhm8Ex9HhxHcWJ0Ri/ckTOMjo43nEytd8/5vFtT4D2kUn5VjixTzGxGcvTwDIcPj/Lqz2eqVqfOwyk4YwsReB8B4YigPWYhpevnEGrZS3agNEG1BLznfcFZn4sL3x/AdduwLMULu2aQUiepx8fHfcEuP2Z0dNxvnbBUEq31Pg+9vb11X3NKIn/nnXfW/P6hQ4cYGxvjlltuAWBiYoIvfelL3HfffcRisVO55JJnVuXMHKtIK5KdZR8gGQhC33rf0lHBUKnccV0/HBmu8PRVKKwtmuF9emXLVBr/L9J14Etfw1q5WovjH91RsdiJfA71P++HmenKNzI9qQVdCLDsUjHy9BREY8X3UbxfKVG3fAYmU9DWoY9PxbWJWkzs+u97va7YIRiCtqjODSDgnRfAsSP6fcW6EXd+HdEenXuB1RK3aDwqFin5nSBPXjCkrLRU6l3z2R9O0RG1iHaWPi6gbZNnfzhVUZVyJggEZ/fBqUDBpVdFaGkN4Lr4Tdl275iZNZMoP4+sMW6V22WOAydGdV38scMZznuPRTBo+dG5580n406xZ0+pb/1SEPqF0hS7ZsOGDXz729/2v77++uu57777mlZds1SoJ0blnjpQEZGrsmOhGBH/ye+XBPPabZXljkqVkpIbBn1rhDderv8XEwwis3qBiAhHsMrsDwna33YK0B6DqVSxb43Qf4VKabGfTmvTEgWpBBzarwco0IPKZPF1U2l454V6kADtqxffm/rGXXBoCPXwvYg/vBnu/64+PhRGhCN6hgI6+VsUeO95nc0lk9WLlJ7fPTPvqkrQn4XMjIOU0rf1pJT88JH0vJUxQuhWwScyLtWXERanJPCWTcUq02qinfojMh+N1N4/9UQG2+/zo1feei0fWlcIptKypn3z8k8zfodKcHlhV4ZUyqWzK8BFHyj7/LuKHz82hRA6ae1V2XjtJYpXLmvsNv89LzXOwls+NZq6UrKOGM2KOsMRyGW1wJcfC7MFcyqN+ruHS618XQf2vqZ/fuBfUXdeB+s21r4fy4ZIK9z2OXAclG2j3vVev6XBrDbB/ZsBAW++rCcCrqO/X2GrCP1J37AJFQqjchOo1rbiQDCpB6bPfQm+fL0elI4c1K0RAHVkWH/vFy+j7rgW+jdXlJCqegnmJbQO4mRxXRgfbXxVpS/m+TShEHzkmjbAplBQFQLf1WMTH69U3PKyRtAflWAIfuVDYUIRxc5/qd3Jq9Eqm7kEHhoT+IVQnmT1rt29CtKp+ou0Ukm9u1Qy6fq9ewAScQchdF18YqK0UEqpUhO08pmXF8mfTYnWak6LyD/88MOn4zLz0qyVkhUliw0uJKrbcGxFe2lOadtaZL0o3dsrzbdO0qVP73nv0RG5dLVBua4fPv15+LNbYab4l+A6MLwPmc3426lVtAn+3C0o14Wbf08fW52k1XeoBf0/3Uji7lt07kDJUuQui/ezcbP25b3nIETtdgu5rFYXT9hrROhLbR3E6SCfL437+RzseHQKYUE0WrleIZV0CYYqq0nb2mcvYCrk4afPzB3+N1pGudh4pZwLYWKs/s1alt4gJZFw/S6ZHh0d+nN/0QdaePlnGSbGdFtkIUr1864r/A6eup5e+R1Az8bP3vKK5Jsw7a8eOLh2Gzx8jy47/Na9qPJItVgPDmjbpEbDMabS0NKmverW9tlz2g0Dut4dpSP19qgud7zudvjWvaU2BpatBb5aqKcn4fbPIy1L3++GQf19L0ew7T8yb32aU4C/egDn9Rdn/2w6DV++TpdvbhiAw8Oob92LuP72ynYLB/fDuo2ov3oAdejAvIPu2eTB18JbVZmMu0Rj1rxRYShUKX5eZH4iK7WdX5ZyscNw2b9ZUeznUinw8/rei0CsG86/IMRPnmqg128NFnsjMQUk47NbDweDuo3yD78/iVQ6QSultn6uKJaWevXzXp28UvDUE5O+2/qrv94xe2HgEmd5iXwzpv1VA4eYntSrTKsGEqUU8pt36yZhgHrnexHX36b/XoviH7/nT2DPz/RfrmWXVnOef5H2wDdugj+4Ge74vG724Tr6r394vx5YPDtESnRLwVTpPgMBaG3Tr5ssVkT9YhIe/F9Ytg2hMDKdrBO9VzE9CW+8VP/nqYR+Jpao6EcvwpFSu4Vv3q0HpKl0qYb/LPPaF4JX/+758t4mGPUGNcuy2Prxdvb8zOHEWKYyIq0KYpWC11/K1IzEW9uq3LYmkJzgpAW+GShZaVd19kDAtkkmXcZHS43OpNKi3xG1/L1nvTp5pRyyWYnrqIoZVT4PkbPsI7qsRL4p0/7qgaOto/ZAUr6DEsDBfXol6ZFhbbEIgTM1WYqipQtDe+H2z2vP/asP63rxfA61YRBeL4qsZ5MM74P1GytFvBzH0Z/ijs7SX71T0D1tvvBf9bOoamLms6JdJ1nLQ8L55vauAxvPA4T/HPx8SHGBla6RF/77lYEgjdieZ2sHyto7NNU/3rIsrvqoblD2zA8m626KUcgza0s+j2YL/NnAZArsgKSQK1YFlyFdvWXgD78/Sc/qAB3Fck0l4cePTVaMp8GQnmGdbf1rlpXIw+JP+2v2eS/vye7hDQa/KFWd8P+396VhUpVn2vd7Ti29VlV3NYsCLc3igguIbKKgRpJMnGTGL99c8xkTJ2rcWNTEDRRwVCTigjgqxEQNycS5vsyX6xo1MaIGUVHBsCuIIsjSIAq91NZLLeec9/vxnL327q5e4Nx/lK7trapTz/u893M/99OoBjqt4mU/w2sB/LMwsOgW8PrRFBxHjCIfmt0fG8G2LQLs/Uz33s6I9hhw+jlqYVcNCo37oUTDYNV+4IXlmaUTTADOOAeQZOqQLQTtMVrbg88CqSSphn71iDFasH40nTjaYwb/v/Jh8Fvvz/nDGcgOlMV0THLOsen9NsQiMVRVMwRqBURCCiQpc+GT5dlz7cXYEx3mDttADYPgEhGFjCofQ8txxZqjcMrqI2EZl/5DFbZt6EDTMdnyXG43fQcABpx/zUkX5EuBTBuH5snOTxutT05i2mYAEP+98mHgMFMzeQHuMWci9dknQEdb+ou0RQ2K5NOtwLIXgaV30N+1K9acaWeTSnzVSBLHPTspu49GgAU3gJ95HlFJmSJIW4SKv9lQWU3VLjM9BACff0zUUluUTjhgxinjoZXko/PZx+oHxoHPPgFPxNM6XC3oYl1Fk5r2ZvHMnvFlsizIlhVSAxVllJ0dZNo14ztVePu1DNcGYHQs26GKocB6h5/vDdTW0WWb7b243HQZc9BlOX5qJbxehmSSVEvv/NXoCXF76GfAQBswYwzRiOk3wGizSCY4mo/JSCT4gPOv6efLG5jgiTgpS8Kka+fPPkw8vUkPz1cupUx+RAMwbxEEKYWaU4fh+OFG4M5r6MpzuekE0NFG5KqW2QPA0jupMSkbbfLAs8B/rQK++NS2OE7qG624q3W0HtqXJUXMo6sT1dqBFqzNkCQj8MciAJih/AFoozG/piTpgyCywuOlk4DaJ2ChgXJ46Cgrl5Ldw/CGXsn+szkWmq2F87kamuqriKp0jMAMARNgtDOYOXsta2cMqB0kIBZVSBffvxNOHWZG0Y6aIEN7OxCoZWhtUjI2PgnqwHFFoc9q/Zukg5dS9Hmau2NdLgEzv1sBt9vg5f21LkTU8YjnTi7HO6+366eCTKexbBu1fX5vX1E8TpDvYXDOST3SpmrPRzQYjo6AISU49KXR3NQWBXwBon7KK6CMO59uH9FA/2WMrtxKH9ARoytV6xTNFoQfuDXz3ztiwIG96by9PQsHjI7XXMVYWaYCakE975yomUP7MrclimJhPwDG1IClav0z2DxYoGb/SrgFkJVeKfAW4liY6z6iCAQHi2hpIqMuf62ITzZ3QOHWr1zMkJ0LAoOnjCMZB9piHIGAmFFO2F+Rq44QaqE3nujkaZc/U/+dtBWp7TQVc5NVcUszdQ7v3BrH5IsqdGWNLyDgsit8uvXEIJPXvDY03T583b5RmzdwX40IcK57+fc2xeME+TwousinOSRKKcBfC9x0N/Dik1btPGOUhbZFKdguuAHKuAngDzxFgX7eIvKhiXcCd19rbAQPP0cbx+//g1wjO9uzn1lzWf5lKsxmwrB64NjX+e/XXoAix4y2KPDrR41TRUUl/TpPG6t7+WT93HW6phXAl/RcJvqGxyKAuVsW0Oshgkh+OL3RTFUI/57tPpxzbPmwA9GwjFOGleOcC+hn+s4a2uA5qAAIZjTyaBAEwF8jQJEU3W74gunl2PR+B5qPZTip9RMU61IJQP8sNNTWAWdPLLf4y2d8mAKcf2E51r9J9giRkIzODhp5mFAHgwNGxp1p3KK2GUtSZvrGotSBDHCudtD2PsXjBPkcyFbkyxSALAZhmpXwyDEQysrBzUM4VM04u/Eu8EW3kNxQStFraBp6leNnbg94tV+VB4jAowsoE5dlKpACKgGpEO0hiNQJ0xbL35aog1GQ7WhLL7raqZ7uQhRprZXVwN7dRkft8AbgaCMAkpIqbo/eawBTTQMeb2Y1U73agCbL4A/9HNzWSasVx+uqq8jwrJtZVCEbfyGWwdnuY26rD7UkwZgb2za2WzJxJjJc9K0qfPB2zPKcF17mxt/Xp2jgtRs4f5oXisIQixYbQXsXXOl+cbilCdiYZciIGVIK2L6xA6kkhygycIXj/bUx/fNNxsl2ecqMdIrNjmwbtfnv9ky+tztnnSCfCxmKfIrbowdsi+eMZhDGFVXjbvJCTybStPPc7aFgrKF+FOnpY22Gj7wgQHn8d+AtTcCj8622vJkalhQZWeUXWcFpc8lnSGKH6FKrW1lomio/3dYeo9eo8gGPrabNRFHIakHDV4fojP5ZjOwOFNmgjxQF/OkHwb9q1D9vuxcQGSpzem5ZMgaXmzJ6bSQia7MZsRWJYtQ9uYJDrvsIAocvIIJDRnCQl7LFsFURwhUObxlHdbWAlrgRwD982zjZpVLA26+1I1ArIJXgEAQgEGSIRXi/VNr0xJoKfY5wiENRgPIKBlnmaVSWfUSgoihIJukEZW6GyrZR2/8OOJx8/4TFQXI0yQCfWUIqF+0XZ+fYTSQhSyWNuam2YqHezKThqpsyd8+2RSnT1k4H7bHMAV5DJmVOPhQygcGOikoKxC43WRbbefvb/x145G7oB2rRBRw7Crz8n/T+ysoBsZre54EvjHVon6GGaAj41ETSqk1VGqfOE3F1gEiI1uKvBZTMGX2PoERmaVqRThC4yg1L8AVEKArH+jdj5D/jBaSkcelt3RBHNE+GnkoC4RAVKBkDIiHepa/7RIOUImorOLgMyUQCkTDZG+hKG7+ou4WaTeEydb1m28ztf+8rFY4T5HNAO+brrfiL5xiSRcbIPkDjdzXPGi2Tt3fU2oqF+MNKqz7+0fkIjznTCCCah4ymqHG5gTPPBX52J9kGmAulokgFxd4chdfZTnxAKgW0ZagLLL3D+u9IK/Dgrda/VVQBP55NSiGANoLKasr0OSfpphmKnO4HZNuI8W9zgYd+Tq93cF/PF1lL0DVtKdIFRERCEhJxMuBiiOt5RN0QEUwAYmEFPr+ASLiwYqp2H87z7+cXXOTC1g8H5i5QrETU5QEu+fYQRCKt+gYrSRzbN3YgEpb0rmS7j1CxXa993TzlBPk8YGrJnmvj5lxu8lIdMQps3mL9S9MoBO726PxxulWurVjIOXnPcADREFIHvqBip/ZrNEsmpRRwaD/wm8eAmC1r9pYBMgcS3Z+1WTAkKXfjlRkjTwcOfpH+9442YOHNNCTEX2vYNjz/BAVonRICAA4wQT8dWXyATJw955zoG0YC55wmcV1AKbqm7UU6f0BENKLAVyPCJbpx9DDxzNGIjJnfoeHUoghs/qADxzuNgOxyA1NmeLFhXfbI7/Yi58YwkAK86KK8SZJor73kH4hD376xA6FWw6GSaCqgtcn6eJ9fpOuFU8bOGIMgMEQjChJxw5XS7CPk8apF7wLRH4Z/O0E+CyzzSW2DP9iNd5EPi1nTrDVEZeKoMxYLVXVNZwcFOV8NZbCHD1Kg35uh6BkLU/erHfaBH72FQm0Lc/0qpBSJv+9ZBlY3WK1f7Kf3Wh0AThlGnbyCCNSPIsqss4MUS1qD1rgJYKp9MpIJcIEKXRBEgzLr1ttMb2rqydOBvXinOSCKIhAMBrHmlUaEQxLAadiHPyBi8sUVmHxxBTa9347mY+QZH6hl2PJh7tT+su9VqRtEJ8kz+3c9VofLTZ2rzSb3SVkCPGUAVAund15vh7+GIRLmkFKk2PF4GPy1Ii64sAybP+hEa5MMRT08Nx+X8cffHoTLbQwGmXRRRVohlTFjHq6dk8+H/jD8m/G83Se9i6NHj/bYc3V13Fum4hqAvNlbrqKcfai1Eg0DS35OXHIgCNy1lOiMVIquAoUXWUAtBcztONnukqdZCgDKKoG4bSNyewxVEAf96s4+X+18/dgwaTPzC5qHjkaXaamarwZs6XM0eCQRB39+uW7oZv7u6k4dVvCoQA29lYlxTlQBAMv80Lq6OjQ1NSGR4Fj/Voz04QAGDxUxZWYVAOiPkySOtX+OZXh2A5d+rxJV1RRl2tuSeOf1/GqUQtElGWQRCNYB504uw7trCpt4IorApVeUw+VyYftHqgbeL0JRFLQ0pV+z3nKGmd+phseDHhv5l+/6KZbK6fXxfycsMhTXtGKfThNkCvbmx3GeUbOtN+8c3EeB3F9LWf0j91CAB2j7tzspaRBdXSuUdgkF7P92kXamJid7gAcoEx7RYHTKyhKwe7vVXtn+Ps30lcutkrBEa/Hnn6DVNu6n09aSVbrmXtt4w2PPAr/x7qJ+uL2ZiWmByB4MGGPweskLvamTNv5wiFrsvV4Gt5uuFVFU0rzZ7Tz1u2vaMfhUFyZNL8fOrT0rsSn1qaClGdiwLo7gYJbTT16DLANv/6UT3jKGZJKDK0AEsuWyJl8ayuS5wrH+zSgCta6Mm3m2jTgXcklpeyuBcIJ8JmQb6KHZBWuZukYR2B+nFmD5kl/o3jV6V+bw04DPdxpDQB5cSWfAu681nocD4HJ6QNc08/0VmQJ8NrTHyEfHDDVg64od0UXP2aluEmUVQFytO5w1Hvi3ecDDvzDGEXLoMlONViGLCdp4pf17ii7EFmMq1h3k20w454iErSe79W/FLLNHNXviTevb0NzEITCgpk4EFFgMt8Ktkkrz9A1X052MP5kAJkytxCebO9H0TWEnXcs8WE7NYlG1eP3tH9QjEmlFMgmsfzNq4eLtn//mD9rR9A39HrVRgYUG+kyJgb0Wk29aWFfhBPkMyFZc44k4mXhJKWB3LM1MizFGAb2liQZ2aCcBc1emORBKEunfR46xXfnqRWnPZIsJov0dVT6qPdh72GW1oDtyDNjN86mYmojTd6GNGHR7wASBbhs5VnW2VKdONZLqSXG5ISTixrQuAK5RZ0AqUg1TSFNTTyDfZpJIWMf+AQyJTo4wZMTjClwuphZkRUy71GfJOFMpBW++bJyCqqvJibGvUEyA93gZZnynDOte69RZOq+X4YLplXj7taiuiw8OZpAlro8vyAoGXDCdgrMoAl6vC4IgwONR4A+IiITJ1oAKstYGtXDI8KIPt2onqa5fE6II+GpEJBMSUgmObRvbC944ikFJg/yaNWvwxhtvQBRFTJw4ET/5yU9K+XI9iq4U1yxUDFeIa7d7zNePohh+aB8FrGjIuH8+tOfmWwcMKqrIRXP+9Zlv5xw4fNBQNr34JAVyjafnCjgTdDUOY8zYDH79GKmQ7rkOiipl1dQ3gS5w8kBhTU3dRb5j/cebrJRXKkXNTYrM8fZrMTAAdUNdmHJxpf582vOIImWe0YiMqmpqHutflbjMYAxqwBXgKaNNzVPGwLkAUbS+gVAzLygH8vvIiEwQ0mmTcFhGdTVRX++uiVmK4ILA4a8R0ZyQ9M/OfpIq/v0xTJxWgXdepxNENKyUhA4s2aW7a9cubNmyBU888QTcbjcikX5MM6jI1a6u16fPmgAc/hKoH6Nzvjo0Tj7SSmqZe5aBBQfRPFXTyYBzDv7MQ8bgj0xeMlX+dJ34iYRQS3bjM38tnW60wH1wn7XbVyv27o4Bi+fQEBVwcvVsj9FJy9aUZldD9UfkOtbHImpm7qZDXyoBuMsoN+CKxlTJSKUU7Ph7p87zXjC9HGv/QuPrRBexfT3RWRocTEreZGE10ILgchMjGWo2DrbNx2Sse62dRG5lNLtVEDgUhVnKVkqOTYsx0sSDA5Gogs0fWDNms61zS1zRLx0OGgYejZAdwaTp5UgmKRZ8sLa94OEvOd+ziyFQ69K/r1LQgSUL8m+99Rb++Z//GW63GwDg9/tL9VI9gnzKGP22+tFgS36VOWiYOXlFAR6dD14/GspNquRSLdyiLUq2CMaLW5+nygeI/XiOZCGKmlzoaAMevSfz8/hqwO5/ioaYAKqjp20zqKymKNAWoeC/09Rp7HLTBqvNwO2pMY8lRD6FhShSRptIyLrdj8cLBGpd4IqC5uMKeZ4nObZtaEc0ypFQed72mEHzyBLQU3qt8VPKsOOjOFq1IM9ob46Gus63SylgwtQKbFjXqauINCQT9BW3HJex6f12TJhaBr9foJm3APTWCPW/bo9RgOYcmH5ZBd57swPgtHHY+W+zjoyDlDZ+U2Ma5xK2fNiJqErn+GtoCEl3A3Nv0IElC/Jff/01Pv/8c/zxj3+E2+3GNddcgzFjxqTdb+3atVi7di0AYNmyZairq+uxNbhcrrTn41z1arEFaR7vJL/xcAsEUUBddZXOt6fdVleXdbAF//cVUMKtaL75f1PKtCsE9u9z4Rp9JnzzH0H0kfmQDu4lGkJU7erM5f5qP5jbDd7a3P1gWir0xJrMw040iC54zjgbgYbRVDSNd+L44f3W2oToQu2v/gQhlTQ+Y21NTIDnvEnw37WEqDatWaqsXM2S6XrIdg2UEpoyw67K4Jzj7de/QUtTHMFBZfjW94aomSHT1zxo0CDM+sca/N/fHgTn9JZPHVGOb3//FDDG0Nkh4dX/bkS8kyMW5Rg0uBytzQkoCvDRe12wuSgA778Vt54IOBDJwoQV04n6wd86MOSUCrQ2J9DZaej49eyaA03fyPjbq+3G39XX1y4nzoF//OEwvPLfX4Grdg57PrGqaoLBIDweES6XC0OG1OGU4Sl8/VUnwIFThpfj0u8OhcvFsG7NMbQ0xVET9CLUkkC8k2iy/3X1CPX7KUxl01PIFNPyPqY7L7hkyRKEw+lUw1VXXQVFUdDW1oalS5fiyy+/xIoVK/Dss8+mfSCzZs3CrFmz9H93RdeeDXZNad5sfXgDICtQhjeQW6FqZpXrtkxQ4p2WK05pbUZS+QzNi+YAe3bRbYEgjdTThlkDdJ5e8ivw1U+RTcGJMKBTdAFjx5GiyPwr40hXD5WVI/mTuWg++hUNFleUdDWRLKP1sYVg8xaRwsZsCjeiAamb56O1vQO8rd0wjVMHswz2+9AUjeX3n+9h5JLKEVXQgXgnh6x0YM3Lh3R6YPLFlairq8OxY830WZg+vuamThw71gy3W4CiKFBUvkJROM6bLCKVqtTVIjoytD3Y84hClS/FUD7FWA2kkkC8M4GZ360EYwrinTSfdfvGDrQ0pd8/W77xwbpvUDdYQDhEs12/OWpUrYODRUQirWCMIRgM4o1XGxFulRCsEzBxeiXcbgFR9bc3fooLslylegoloSgM1X6mP7630es6+cWLF2e97a233sLUqVPBGMOYMWMgCAJisRh8Pl93XrJ7yGEulatdvdBWdkvD07gJagBS2+tkSQ10oF/WsHrqbjUFePHciVDKK8DmLqRmqQU3nBjz2o4eRrrmPoORSnsbsPAWKKJInPw1c9WTju1xjfuJZ1d99+1WElzrUTiwlzbKaAi4+1o0ud3gw04zZusCfT5AxGJJa6IHoErq1v71azQd64AvICA4RCDZIyejsm0fdWDyRRVIJLh+mUgpauTxeonrDbeqhUI1wDNGenDOgECNiHMnleGTzR2IhMiZsrc6YHOZnkaj9N53bo4jpHb60m8uc0TPdFKIhGV86x99SCY41v3VECzUDRYxZQYVpyWJI5VSdFvnGOMQBCHtt0/fVe+orEqBktE1kydPxq5du3D22Wfj6NGjkCQJ1dXVpXq5wpDHXCqXosZ+m71Iaz8lsLkLwVJJ1VI4Cv7g7caTVVYD8xaTZ7rmaClLkHduBZ5ZAn7T3UQqlpVnNv8aSJAlkormmummw2RKpm2QWgFatyoQ0r47xphl2Ij+PWhndbUOwjmnCt2IBnVISe/w9bnkkWZO1nCgpPtxznH0SAcUGWg+ruDb/1QFRWF4/29tVPQLUWHQLO3L9rzJJPSOWW85w0WXVWLn1nZ8uLYd1QEBF11egffebLcE+YK+Mhuy9cPZwZXsNE4yDrz7RgyS5bSQHuA1/3kpldmLXhTJSlhrEHN7gCkzK8CnsWMtAAAgAElEQVQY009Wg4ak4AsIAPJ7vfeGyqoUKJmtgSRJWLVqFQ4dOgSXy4VrrrkG55xzTt7HldrWoOhJTxmgKAr4M0uoeKra2SKZgLJoNp0SfDXA/U9BUEf6KYoCftdPDeqhOgDhkd/QgJG7fppeWHS5jf/XTDgGislINjy4EnjwtsKtGqr9OgUDOQV89gkABowbD3b7A/S5ZqDelHgn+OLZZAbnr6Xms68aAUUGc7nAR4yyGJr1JidfSBZovp8kcbz1agyKTD/RQUMpC9XH1PlFhMMSknG12Oil7NysHLFbGIdDMrjCAWZVxnR3YAdQ2mZsj6okMq9x0GARTcfpevKWM1x4aQU2vkPTnrzlDN9SR/gpioJEgkMUqdZh3vAqKkWVGuoZG4NSo1/ZGrhcLtx2222levouo7vmUpzTYG49A9fsbD1eCkjRMKU/C26AMu58CPPUjF50aQvQM0ghmSAttx1aeiO61ADfD4uvxeKh29U0zxbkmUDUzAGbS6VWmG2PmVJDDuzZRU1ogmBIK9WNm3u81vm6p40Bm3uffqIa5Ks2JkOVmKKxo9As0Hw/lws4ZVg5vmqkLt9IWIaiUIYuSRxbPqRpRowBdUNFXHBhpaUQaK8FXDC9HJvXt6MpgyVAMQG+Jigg1JKedJithtwe4Pxpbmxan/sk6nIDF88qw8eb4whlKMdppq9MEBGLKPB4aERMoFbA+dPKseOjOCIRGYEaEZVVIvy16jxWkx88Yww7t3SkUVfecobgoLKsxdO+tgjuKfRjnV4/RTJBGbym1RpWD8XlJj235icPqNbA+8BjEcrYFZnuX+UDVDsE7vYA9Q1ESVRU0S/DfDHJkvp86SoUVPqAHjXRNcFb0fPPKUuZz+bjJtB7tyPbxiZLwG8ep89Up2PIUli3MJBSQGU12I13EsfqLaP/qiqbXNC8iXr6gKupa4p5XsYYLr9iiH6w4wpl5Np7aG3ietCaOK0CbreVTzbXAiIhGakUswwZcXvoUrIjOJhu0xAIWv+tBXhLfsKAgN/4g5QCtmywft/mA6q+Rgnwlrlx4aXVqBsiwuO1fj8Xz6qEKIho/obmr0IAZn63CgwM773RBjCOy75XbQzQVhQk4xxNxyRs/rBDD9Qa755MQD/5zPxOta5msn8v2gb5zutRbP6gvcevh97EAGSY+hbc7aFhIQBdoV81AvdcBw5G+m/tYlDlkfzB26nLVcvIBRGClAIXBFJ5fL7T5M3Oc2ftGr+sTWXS/67SOaJL1eh3Qw3NWO/50p81Hrj5HmDR7My3+2roM47bnBIb94O1xyyWwkgmwF9YbnjfnDY6vVktD4oZ7Vfs83bViEqWjdqzJKnjfW2pGUMOfX0NjRGs8gkQBAk+n4AWSYHI6LIRuFU7L7qByRf7IIrQKQ5NwRNukbDBNEPV54duIyCKwMSLvPh0G9B0nHTu3ETdCGLmPZ5zYNuGDoDRDNTKaqvBmiACoZCxwlQC+HhTh+67EwnJuk+RJNFzaD8hzYNGq4lwLqlKW7IV9niAdWuOoelYR9r30h8sgnsKA3TZfQPdtqDxAKljvmq0jvzTxs+NaKDgpDksNu43NoYRDaQGSSbI2kC78jMFd7u+Tft/u6xQOz30BCHamxnLkYOUImZbN1dIz67drtcmOHhFlXE6kmXKtLQsPlBLnv/FBugSjfYrNGBkowc0XYn53bhcDHVDjU5Jl4tlfrxCjVGJThlvvmxs3goAZAi6ogC17V9Aebmxrq0bOhFqsX5PZp8YWQLeeb0D/+fakTjc2IR311glxva8w1wDiERowhXnhhGr/t6ZgupqAa0JRV0LTczS4A+kD89uVs3Y/AGyFeacWewJNBthWQZamuJ5VU99MXy7J+EE+WJgti3gnKgWxtRgQ6oPduNdAAC+eA49RqUT0HgAAAeOHKSNYs59QP0YoG17dpnkqDOB/XvyF11dbpXaGWBHylgEaG3Obqsci9j4BPX9CSJl7O3qyUmba2tSTuXL4jMW4Esw2g8oLGBky/bdbiEtmAMUxKfYBkXbHy/LQCSc+9rRaBhZotIHExhkmfT22mahmXNZDdLSkUwAa//6DY4dze11UFsHTJlZia0b4oiEaRKWFpjBgdpB5HoBTlbBZni80CdnacNTzJbMU2ZU6bTYto3t+NurJJ/UfH0YE/STkChyBAeVQVE6cqqeBjon7wT5AqGPCRuhTnTqaKNg89BKAIaMT9dpa8FC08NHQ+l+KvMWUuPUc48CRw7ANXYcpN2fAJ1q0PryM+D0c4AvdmVf2Jnjga8byW53ICJTMVYL7LLKT1RW02fNYLUq0H546nGdFTiWLxstU2g/RLEoJGBky/btwTyzhpuUOPbHa5vL8a8zn5QsI/M+opZ9f0BI87UXRQqs5lGDxnMwpJJUG3B7gNbmuEVC6fEyJBPW5CMWA7ZtjIMxaGOPERwioOW4AgZAFEW4PXKaL04gqHn0EQ+fqWBKG6NB3WhryWQfTDWPoTh2rDnj92L+fAdyEfakDfLFSCktQWH4aRR0Iq1Ew/zmccrOVW08YG2e4m4PqXF2bTN48/rR+usKZeVQ3G5AECAwBtz7GA3q1pCPXz9ywLAqliX9RIFUwmi+yovsjSZFo1ChtIZFN6e7S1nqCpw+M9XPRpM+AoCiNZyNaDC+x0LolTxNcaVQ3uRT12TL9gsNLjrvrDZVUYFWwOSLKxBX/V08HkCSBLhcClIpho83tWP9m+06T+2vceH8aeV474023eNcG0wy+eIKbH6/HaFWGVIKelC/9HuV2LaxHeEQfYdmuoUxIFArQpYltBw3/p5KUNAFY7r74oxvV+n6/2hEQSAgIhy2BvqI6ovTfFzRLSLyfR46dWPL1LXPFUDO59Hu29dzWruDkzLI5yuwpW0AtolPqG8ADjPi2bN0T+rBgnNwzSqQbgBuuINeR1NxqEO+kx9vBvbYZrvu+yz3m2mLAvv3Aqc1AEcaaSrSvEXgigLc+W8GlZH7EyngPgWivLI4S2Ttl8YYZenmwSAa2mPAf64Em7fQQsOwuQtpAz18AHzVL8GzjFvMaiQH9BsDs0zZvuZrk6kwmOnxky6iQBwJSdjyYQcmXVSha+P9NSKmXFypBjQBnCuIhhWT7YHhqFgdEKBwGYpMk5L8NSImXliJyTMqVSpH0btqZZlUPuZ9XRCAS6+ohMcj6gVcLYALguYhJyLSSt89Vzg8HtL4R0xzbmUZ2PRBFNEwtU1EWuhK5Qrw3psx1NSImDyjMuPMVTN1AyCjtDQckjF4SArjp7hzBu2BXoQdQEvtQeTI5DLOd7UFBa2Zhrs9Fh+UTMGCJ+IqH2+F4Wo5yhjqnUp2rSOlI6Y2CwH4/BN6zecfLzDA9zDyBXiNdrHbFXBOv+S7fwn81yp6P5pMVZZo8pOqhdeDdyoJfvhA2veoKIoe/DNt4ozRcBe0RYEqX8FZWU800uWCPdvPVRjMBEVhiEYocIdDNEyk6RsJigI0JyTdeZFzju0fdSCVVJuoPNDH3733RoxoFMaQkrhu9/uO6q8OTjSItulwrqQd3Digmn9RZ2moRdK/Sg7KqidOq8C7a4xrJZVilgHm2jO5RBGiqMDlElA3hJEUMsGRSgDHv5GxaX0HpsysyBroM01akiSufy5HD3fi7ImunBOZBnoR9uQM8rkyuSzzXdO4Wm8ZkRw5OFzOOTXntMcAMPqFjptAWZr5NRY9CTz08/Qecru6phAoqiLl0P7c9xt1BhV1i4GZiqnywd0wFqld24pbY7WP6hqff2IUnBkjSkaSgPvn0vNVVquSUEWnoLjbA262jphzX9r3qCgK+H88QMombV1ao1S8U9c7mzdnXoBUslTyylwQRWQtDNrXpnW1apOGkgmOHR+1Wc5omlZfURTdCsFbznDJdyuxfWOnZUQgQFm320tfk+adDs51bx1ZVjdM22VaN5gCvCwDoVZJL9hqGoRohIZj+GuocYlz6wAOAHqmnUrQKSEGBZd+j2xRtn7YhiZVQtl0TMLm99sxZWbXJirxAk6xA70Ie1IG+ZwFtiwbQDauNieHm0wQby9LQHUAWPAYWN1gCvLmk0G1H7xhLGWrqRTQ2UG/mvKKzJlxZTUwdBjw5efpt3EF+N3TuRU5glB8gAdoTaPOAI4eARQFqcb91NwFpnam2rJzbdC2Zvbtq6GxfnPuo8Ab7wQevoPorspq2kA0uWR7DDjjPOD2f4cgS9AMSMybI0slLcVWAJTB795Bz8GIUtM2h+YjB6AMbwC74c7ipZIlklfmQr7CIGCiHlol+AMixk8tw9o/U+bc2kwBNxohD/Qdf+/UbQ1SSboM/DUiBEGwNEl5PAATGfw1Is6fWo7tf+9ENCTDZ8vkadNhCA4WdN26t0zQFS+CoFguQ6ZaL3CF4/23YvDXujDjO9WGF4+6cQDqoHKV3vGWQVcWMcYwZWYVNq3vQNMxulYi4eIoFJeLoW6IqHrXVOTl5LXvYiBRNGYM0GV3H7mCdlcVFmnHedtgbzx+r16gtb8Gn7sQddVVaIrGKJgl4pTdm1HlBxpOB268E/jNY+ZFUwdsW4Re6/NPchdsMxZGCyi+cp6+OVT5qUaxx1bknf8o8NofqfjLGHDmecBNd0PQfPgZA3wB8NPGAI37qBjNOQVoTQ665xMycbvtfjDGoLg9hjZekcHdHipWa1RbIm4dxnLWeLB5i4nWOfQllHALWTgDxXPyfcTjU3DJzReHW8m58vg3MrZ/FDc9Fpg4nagMzjneXROzDOPweKlT1uWi5qAQl3RBvt8vYPJF9Fi7VNNeNyCqRFFf09ChKwqD+efDOeCrZojG6DQQDcn6a0dMVAjnnPx1QHnCpd+rtnTzCoKAKTPV+kNYRqDWVRSFovH1sgwMGVLXpZGQAwknbZDPha4oLLId54W5C8FjEfAlv0ijgLJtMqysHNxbBqW+AdgV0m4A7n0MrG4IWCoJ5fBB4+9nT6T/mimKTBRKtT+9kcp4B9nfnOiitWbi+IeNAI42pjc0PTof+sYhuoDD+4EXn4Qy+15A58vHQJ9dBwBzFwLPLKENQ5NwNO7Xs2aWStJsV5KDkNLG/BmaA/GIkWDzFlOAU/8uiAJl8t6yguWWGkolr+wudHnjN7Spx6KKOs+Vsm0tOHJuqG+4wlXli0vPjidfXIlEguvGXZGIgmQS8Hp5WhZrrxuEQ8Z3n1QtkKeo0suaoAuhVklvdorFOAI1LpJrqtm55sWjwbw5MAaL/a+hNqKMvqsUivae+sv3WEo4Qb6nkOU4zxgDqv0GPVM/ijJ+zi0XrmKiE3S+96Z7gDuuMQLeYwvAR46lRqrTxgAH9wIjTgPmLgSTUqSoSSaA/3zWyIg1VPnoMZ+rBdozzlUz5+3535u54cgMxoBb7gVWP0WngzRfWtVygDHSvx3cCzzzEK2Nc+DQXgrw0RBwCGRV8NUhW3MYpw5hgIL4yDFZC93ZArH297rqKsOgDCh6I+8LY7N8YIxh8oxKbP6gg7Ja0/Bpc/Az88qMkYTS47He7lVdLMPqRrD+zSgCta6cqh5B4Jb8QJE5orom3Qjg2za26zSPfX1aIViTKE66qCLj3NNMUsZCqJaTHU6Q7ynkOM7rGX0iDv7CcvDFc8A15Q5AAy7MdIK6QQhl5YYWvD1GgfLQl6q5B6e/fb6LvHMEEfqEpOEjyZtVa6qqqAIeWgX89knyvakfDXbrYvCmY6RTN8MXICdNM0adQY1Z9tOBywX8dgUF8Y42ACy9uam8EmgYS5tLW8yYjMUY6dsP7qP7KTJ4ZbVBb2kDuQWRvOCr/QVl07loOFZWnnOi10CFIAiYMsNaGMwgNgFjDKLIsfmDzoyab0tGr06VytREZIYs2/ZkRvNotcCsKVw0eiTT+uwSRc1p056lD3QpY1/B+Yh6CPkCECNdmqVwqAV9HNwHKDKE4CAaM2gq9grzFtH9nl9OhVltI2n80sjUNQpGkzlotgsaOtqAe64zfo2HDxB//cffWN+Ey50e4AGa7FRRDd3wQwvqqRTRKYDx3BVVFOi1rF5gwI13kWom0mr4+9Q3ADfcRVp+AOjs0AuplveryOAP/Rxc9e3PFMRLLW0cCCi0MGgPlJJE1I0WTPWMPkMmnQ3mrb+sjGHihemZf671ZZIoZrr/QJcy9hWcIN+DyHuc93ipwAiu/heGFw4Ads4FwOx7LTSOxt+zeTYVieZ7A5AKR5unJgiZLQ7McsURDVS4/VR9vC8ALHwS+OWdmR/bETMeKwgU8LWftiQBDWNMKiAbtcMEMEEA12iW+lHk7+PxgkfDaacDvSYxT61lPPRz+ny0E0yGAN/b0saBDMu4wRrRQqNo8kVZRkbKJxM0pUqzWhOoG1yeRqHk69otRqJ4/rQK/XWd77kwOEG+16FxmFwdNDJSz7zlXduAlQ+D33o/AKQHL1PDFrvxToODd3uA55+grHrosPRAzZjBjZ91HnDj3cDiOQZtctposEAt+GljgV1babNwuQAw61mcc4oAMVO23x6lxqUzzwWuvR1YeicgRVXP+2qgYWxaoVN/b59uV6Uaav+AuZtVq2Xk4OAB9Im0cSDDHFA1xY05q7f71uQLpIwxTDV1lg4dOsiiVinUEiDfSSTT8zgoDE6QLzEsw7013XykFWhkpA6Zt5iad3bvAMDJzCyZoMdpk48APXhxzqE8+7A6A1UhTT1AhU9FBvaEqcja2U4B1OWmwRw33GnIOpMJki4eIt8XNm8xFcBuuAN47hEK2pJEwddXo5qsHaBsXcu8zR0wsgQcOUQZuF5gHg124526aRsAq9TxoM1mWVufCQUpWvqhRUF/hxZQNcWNRn8AmTnvQjJxrWPUfnt3eHTz6zp8fNdRso/p4MGDeP7555FMJiGKIm644QaMGTOmVC/XL6FTCZqJ1rxFaQFJYAzK7Q+AP7sEwtFGKMMbDLsEbQBG/SiDpknEaUPIZk8MUOBXSGaIM84hXb6qlTbbKbCHfwWmbRzPPEQbUHubNZCPaKC/t0XVzPwM4Mh+oH403KKI1M6t+vth3jLw2fdSF05tHYRspKnHS5z8zlb9dXJ5suTKzPurtHEgwE6TAOmcd3fNubrKo9tfd9JFFZbnEQROecgA7EDtbZQsyL/00kv4l3/5F5x//vnYtm0bXnrpJTzwwAOlern+iWTCyMajIWDlw3pzjjkgCYIAfuv9usSPxtjtK34ARmU1bQrtMaPjdc8uWkdZeRq1oXP/zzxELpn0R2qsSnTS6x85QE1WAHHtP7qJrAkA+INBGiqcoHlqiiwD86+nQnC1H8rjv8vqKcLmLVaHoe8HThtb9BQn+/M5FE3XYKdJ7Nx4JgvjYjLorloC5FLcaEPJB6orZG+jZEGeMYbOTjL87+joQE1NTaleqv9CH+4douz48EFSkNgCkqIoJBMMBgG0WYdR14+xcPEAiH45tA+QZHJs1FQ2oot8cF5cbtgMc3VmqbfMSm1oen27gZrooix772719UfTRtUWoV/psnvofp3taKZpyJT9gxNN1Bal22MR+n9fIONHIwgC+G33lyQDd9Q2XYc96PeEoqUrlgC5FDeS5FA3xYDxEk2oPXLkCJYuXQqAgtjDDz+MQYMGpd1v7dq1WLt2LQBg2bJlSCa74MKYBS6XC5LUAyPxugFFURD55d2QDu6Da9QZCNz7qCXwKIqC5p/9E3gkDMEfQO3K/0brbVdDaWkCq61D3bN/hFBeAc45wo8sgLT/c4gNpyNwx4PE0cc7EX5iMeRD9PyMMSR3bCI3S0EkD3aXqL82APB4JyIrHoC0fw/EhtMBAKkdmwBwiOdcAPnQPiDUAlZbh+Az/5ceEw2j5b5b1JE9WaAWSxGLgvn8CL74Z4i9rHMzf072z7s/XA/Foj+sWTM2K0TR0pPrzfa6mgVzS1McwUFluPyKoV3ezPvD51sMsq3X4/FkuDehW0F+yZIlCIfTddVXXXUVdu7ciXHjxmHatGnYsGED3n77bSxevDjvcx49erSry0lDXV0d0Ql9jFyZpRINg9/1U71VH4+vBv6wMk0SyBNxKItmE9USCEJQ+XTz83POwRfPNoZvVgdUSWWr/hh4vFZpYiBI061+/ZiqeedGMfes8fT4Q18a0s+D+zJ0tqqo8gGP/hZYtRQ4cggwa9t7Cbk+p/5yPRSDgbbm3lpvT01qOlE+31NPPTXrY7p1yMkVtJ999llcd911AIALL7wQv/71r7vzUgMaOTnjKh+ZfLVFwHx+8HgcuGWB7ryoX8B5Omr1ASVm/bw2MUptotKteg/uIzWOv5YKpoyRnUAsbKhmArXAT28ll0iNw1+yijaU+2426Bsweq7yCvr/lQ9T81QObXsh6DLl4qhtTgoMZFfI3kbJPqba2lrs3r0bZ599Nnbt2oWhQ4eW6qUGNBhjpAU/uA88EiKbAZcLyrN/gmgKbrlUJOaAKMyjjlEABvev3sYTcdKmSyni25e9AEHlzLnZLVObo1rlS3N9ZO0xcJWCYf4a8PuW02tofvi7dxBlEwh2Och2p8HJUds4cGBFyYL8zTffjNWrV0NRFLjdbtx88835H3QyIpkgDbq5wUiSSIY4aIjlrtla+s0BEbPvpWKqxwtme4ydlzNr2M0zafU5qskEFM31EQxYuZQmMXE6BbhHnwlJ3SSU+lHAp+oAEUEEW7yC6gFdCbLdbHBy1DYOHBgoWZA/88wz8eijj5bq6U8cmD3nIyEAnKQCtXWFPd4eEO/6KalaGINy7iQI8xYZBlTeMvBxE4zJSvYOU3Xalb4pmF0fh4+kzUjlutniFQg0jNa7G9mti8GfXULNXOoglC5n0Q7l4sBBj8FhtfoYZnqhZshQtHzxOVBVrevL83LT5oA4rN7wo9EGhNuGiwvzFhVMZZjXpjdoAXoQB9TmLI9X1/r3BE3iUC4OHPQcnCDfD6Bl0aIoAv/vBbIdVmeYmmeRZuKmzQFRcbmBu68ljbrqSZPJcz0TlZFtMzFn+NzmPxN+ZAGUvbuta+shmsShXBw46Bk4Qb4/IRG3Ui9tUcu/udpZmi0QiwCUJ35P/vQeL4Sy8oKy4EILnRZ+PxGHtP9zxxjMgYN+DifI9yd4y6xcdJXP1KE6Gvz5J8Ab9+cMxAJNZy7udbtS6PR44Rp1BpKyTBp6hzd34KBfwgny/QiZuGiNIqFGpzmlyZy7WOjkHIZtci/BsSxw4KA4OEG+n8HORVsanUqkOOlSoTOZgHxgj26b3Bt0jTMgxIGD4uEE+QGCUilOLJlxMUFap2uU3pM5OgNCHDgoGk6QH0DoacVJdztLA/c+iuajX/UedeLo5x04KBpOkD+Z0QOdpd3xgS8Wjn7egYPi4QT5kxkDMDN29PMOHBQHJ8ifxHAyYwcOTnw4Qf4kh5MZO3BwYiN9AKcDB30Ero0qLM2wMgcOTko4mbyDfgFHA+/AQWngBHkH/QOOBt6Bg5LACfIO+gcGoNLHgYOBACfIO+gXcJQ+DhyUBk6Qd9Bv4Ch9HDjoeXQryG/cuBF/+tOf8NVXX+GXv/wlRo8erd/28ssvY926dRAEAddddx0mTJjQ7cU6cODAgYPi0C0J5YgRI3DXXXfhrLPOsvz9yJEj2LBhA5588kksXLgQL774IhRF6dZCHThw4MBB8ehWkB8+fDhOPfXUtL9v3rwZ06dPh9vtxuDBgzF06FDs27evOy/lwIEDBw66gJJw8q2trRg7dqz+79raWrS2tma879q1a7F27VoAwLJly1BXV9dj63C5XD36fKWGs97SYqCtFxh4a3bWW1p0Zb15g/ySJUsQDofT/n7VVVdh8uTJGR9TTMfirFmzMGvWLP3fzc3NBT82H+rq6nr0+UoNZ72lxUBbLzDw1uyst7TItt5MjIqGvEF+8eLFRS8kGAyipaVF/3draytqa2uLfh4HDhw4cNA9lISumTRpEp5++ml8//vfRygUwtdff40xY8YU9NhcO1JX0NPPV2o46y0tBtp6gYG3Zme9pUWx6+1W4XXTpk245ZZb8MUXX2DZsmVYunQpAFLdXHjhhbjjjjuwdOlS/OxnP4Mg9L4X2oIFC3r9NbsDZ72lxUBbLzDw1uyst7Toynq7lclPmTIFU6ZMyXjbD3/4Q/zwhz/sztM7cODAgYNuwrEaduDAgYMTGOIDDzzwQF8vopQYNWpUXy+hKDjrLS0G2nqBgbdmZ72lRbHrZdyZ0ODAgQMHJywcusaBAwcOTmA4Qd6BAwcOTmCcNFbDf/7zn/HSSy/hhRdegM/n6+vlZMUf/vAHbN26FS6XC0OGDMGcOXNQWVnZ18tKw44dO7B69WooioLLL78cV155ZV8vKSuam5uxcuVKhMNhMMYwa9YsXHHFFX29rLxQFAULFixAbW1tv5f6tbe347nnnsPhw4fBGMPs2bNx+umn9/WycuK1117DunXrwBjDiBEjMGfOHHg8nr5elo5Vq1Zh27Zt8Pv9WL58OQCgra0NK1asQFNTEwYNGoRf/OIXqKqqyvk8J0Um39zcjJ07dw4Ij4rzzjsPy5cvxxNPPIFTTjkFL7/8cl8vKQ2KouDFF1/EfffdhxUrVuDDDz/EkSNH+npZWSGKIq655hqsWLECS5cuxZtvvtmv16vh9ddfx7Bhw/p6GQVh9erVmDBhAp566ik8/vjj/X7dra2tWLNmDZYtW4bly5dDURRs2LChr5dlwaWXXor77rvP8rdXXnkF5557Lp5++mmce+65eOWVV/I+z0kR5H//+9/jxz/+8YCYNjR+/HiIoggAOP3007Mau/Ul9u3bh6FDh2LIkCFwuVyYPn06Nm/e3NfLyoqamhpdkVBeXo5hw4b1y8/VjJaWFmzbtg2XX355Xy8lLzo6OvDZZ5/hW9/6FgAy0eqPp087FEVBMpmELMtIJpOoqanp6yVZMG7cuLQsffPmzbjkkksAAJdccklBv7sTnq7ZsmULamtrMXLkyL5eSnCzZjcAAANUSURBVNFYt24dpk+f3tfLSENrayuCwaD+72AwiL179/bhigrH8ePHceDAgYJtNvoKv/vd7/CTn/wEnZ2dfb2UvDh+/Dh8Ph9WrVqFQ4cOYdSoUbj22mtRVtZ/p3zV1tbiBz/4AWbPng2Px4Px48dj/Pjxfb2svIhEIvpmVFNTg2g0mvcxJ0SQz+WU+fLLL2PRokV9sKrsKMTZ83/+538giiJmzJjR28vLi0yq24FwSorH41i+fDmuvfZaVFRU9PVysmLr1q3w+/0YNWoUPv30075eTl7IsowDBw7g+uuvx9ixY7F69Wq88soruOqqq/p6aVnR1taGzZs3Y+XKlaioqMCTTz6J9evXY+bMmX29tB7HCRHkszllNjY24vjx47j77rsB0BF4/vz5eOSRRxAIBHpziRbkc/Z89913sXXrVtx///39MnjaXUZbWlr63VHXDkmSsHz5csyYMQNTp07t6+XkxJ49e7BlyxZs374dyWQSnZ2dePrpp3Hbbbf19dIyIhgMIhgM6jMkpk2bVhBX3JfYuXMnBg8erIswpk6dii+++KLfB3m/349QKISamhqEQqGCRCQnRJDPhvr6erzwwgv6v+fOnYtHHnmkX6trduzYgVdffRUPPvggvF5vXy8nI0aPHo2vv/4ax48fR21tLTZs2NBvAxBAJ4/nnnsOw4YNw/e///2+Xk5eXH311bj66qsBAJ9++in+8pe/9OvPNxAIIBgM4ujRozj11FOxc+dODB8+vK+XlRN1dXXYu3cvEokEPB4Pdu7caZlR3V8xadIkvPfee7jyyivx3nvvZZ3pYcYJHeQHIl588UVIkoQlS5YAAMaOHYubbrqpj1dlhSiKuP7667F06VIoioLLLrsMI0aM6OtlZcWePXuwfv161NfX66e6H/3oR5g4cWIfr+zEwfXXX4+nn34akiRh8ODBmDNnTl8vKSfGjh2LadOmYf78+RBFESNHjrQML+oPeOqpp7B7927EYjHccsst+Nd//VdceeWVWLFiBdatW4e6ujrccccdeZ/HsTVw4MCBgxMYJ4WE0oEDBw5OVjhB3oEDBw5OYDhB3oEDBw5OYDhB3oEDBw5OYDhB3oEDBw5OYDhB3oEDBw5OYDhB3oEDBw5OYPx/sT+2EsW8UekAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 生成符合正态分布的聚类数据\n",
    "from sklearn import datasets\n",
    "x, y = datasets.make_blobs(n_samples=5000, n_features=2, centers=3)\n",
    "for _,c in enumerate(np.unique(y)):\n",
    "    plt.scatter(x[y==c, 0], x[y==c, 1],s=7)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**在本开源项目的章节中，我们要重点介绍回归和分类的集成学习的问题，因此我们在接下来的章节中不再介绍关于无监督学习的具体算法，后面的内容仅仅涉及回归和分类问题。**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. 使用sklearn构建完整的机器学习项目流程"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "一般来说，一个完整的机器学习项目分为以下步骤：\n",
    "   - 明确项目任务：回归/分类\n",
    "   - 收集数据集并选择合适的特征。\n",
    "   - 选择度量模型性能的指标。\n",
    "   - 选择具体的模型并进行训练以优化模型。\n",
    "   - 评估模型的性能并调参。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1 使用sklearn构建完整的回归项目"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(1) 收集数据集并选择合适的特征：        \n",
    "在数据集上我们使用我们比较熟悉的Boston房价数据集，原因是：        \n",
    "   - 第一个，我们通过这些简单的数据集快速让我们上手sklearn，以及掌握sklearn的相关操作。\n",
    "   - 第二个，我们用简单的数据集能更加清晰地介绍机器学习的相关模型，避免在处理数据上花费较大的精力。\n",
    "\n",
    "如果您对具体的项目感兴趣，我们会在第六章给出三个大型的案例让大家体验。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-14T11:47:24.395506Z",
     "start_time": "2021-03-14T11:47:24.372506Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn import datasets\n",
    "boston = datasets.load_boston()     # 返回一个类似于字典的类\n",
    "X = boston.data\n",
    "y = boston.target\n",
    "features = boston.feature_names\n",
    "boston_data = pd.DataFrame(X,columns=features)\n",
    "boston_data[\"Price\"] = y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-14T11:47:26.933502Z",
     "start_time": "2021-03-14T11:47:26.908501Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CRIM</th>\n",
       "      <th>ZN</th>\n",
       "      <th>INDUS</th>\n",
       "      <th>CHAS</th>\n",
       "      <th>NOX</th>\n",
       "      <th>RM</th>\n",
       "      <th>AGE</th>\n",
       "      <th>DIS</th>\n",
       "      <th>RAD</th>\n",
       "      <th>TAX</th>\n",
       "      <th>PTRATIO</th>\n",
       "      <th>B</th>\n",
       "      <th>LSTAT</th>\n",
       "      <th>Price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.00632</td>\n",
       "      <td>18.0</td>\n",
       "      <td>2.31</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.538</td>\n",
       "      <td>6.575</td>\n",
       "      <td>65.2</td>\n",
       "      <td>4.0900</td>\n",
       "      <td>1.0</td>\n",
       "      <td>296.0</td>\n",
       "      <td>15.3</td>\n",
       "      <td>396.90</td>\n",
       "      <td>4.98</td>\n",
       "      <td>24.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.02731</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.07</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.469</td>\n",
       "      <td>6.421</td>\n",
       "      <td>78.9</td>\n",
       "      <td>4.9671</td>\n",
       "      <td>2.0</td>\n",
       "      <td>242.0</td>\n",
       "      <td>17.8</td>\n",
       "      <td>396.90</td>\n",
       "      <td>9.14</td>\n",
       "      <td>21.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.02729</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.07</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.469</td>\n",
       "      <td>7.185</td>\n",
       "      <td>61.1</td>\n",
       "      <td>4.9671</td>\n",
       "      <td>2.0</td>\n",
       "      <td>242.0</td>\n",
       "      <td>17.8</td>\n",
       "      <td>392.83</td>\n",
       "      <td>4.03</td>\n",
       "      <td>34.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.03237</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.18</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.458</td>\n",
       "      <td>6.998</td>\n",
       "      <td>45.8</td>\n",
       "      <td>6.0622</td>\n",
       "      <td>3.0</td>\n",
       "      <td>222.0</td>\n",
       "      <td>18.7</td>\n",
       "      <td>394.63</td>\n",
       "      <td>2.94</td>\n",
       "      <td>33.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.06905</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.18</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.458</td>\n",
       "      <td>7.147</td>\n",
       "      <td>54.2</td>\n",
       "      <td>6.0622</td>\n",
       "      <td>3.0</td>\n",
       "      <td>222.0</td>\n",
       "      <td>18.7</td>\n",
       "      <td>396.90</td>\n",
       "      <td>5.33</td>\n",
       "      <td>36.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>501</th>\n",
       "      <td>0.06263</td>\n",
       "      <td>0.0</td>\n",
       "      <td>11.93</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.573</td>\n",
       "      <td>6.593</td>\n",
       "      <td>69.1</td>\n",
       "      <td>2.4786</td>\n",
       "      <td>1.0</td>\n",
       "      <td>273.0</td>\n",
       "      <td>21.0</td>\n",
       "      <td>391.99</td>\n",
       "      <td>9.67</td>\n",
       "      <td>22.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>502</th>\n",
       "      <td>0.04527</td>\n",
       "      <td>0.0</td>\n",
       "      <td>11.93</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.573</td>\n",
       "      <td>6.120</td>\n",
       "      <td>76.7</td>\n",
       "      <td>2.2875</td>\n",
       "      <td>1.0</td>\n",
       "      <td>273.0</td>\n",
       "      <td>21.0</td>\n",
       "      <td>396.90</td>\n",
       "      <td>9.08</td>\n",
       "      <td>20.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>503</th>\n",
       "      <td>0.06076</td>\n",
       "      <td>0.0</td>\n",
       "      <td>11.93</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.573</td>\n",
       "      <td>6.976</td>\n",
       "      <td>91.0</td>\n",
       "      <td>2.1675</td>\n",
       "      <td>1.0</td>\n",
       "      <td>273.0</td>\n",
       "      <td>21.0</td>\n",
       "      <td>396.90</td>\n",
       "      <td>5.64</td>\n",
       "      <td>23.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>504</th>\n",
       "      <td>0.10959</td>\n",
       "      <td>0.0</td>\n",
       "      <td>11.93</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.573</td>\n",
       "      <td>6.794</td>\n",
       "      <td>89.3</td>\n",
       "      <td>2.3889</td>\n",
       "      <td>1.0</td>\n",
       "      <td>273.0</td>\n",
       "      <td>21.0</td>\n",
       "      <td>393.45</td>\n",
       "      <td>6.48</td>\n",
       "      <td>22.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>505</th>\n",
       "      <td>0.04741</td>\n",
       "      <td>0.0</td>\n",
       "      <td>11.93</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.573</td>\n",
       "      <td>6.030</td>\n",
       "      <td>80.8</td>\n",
       "      <td>2.5050</td>\n",
       "      <td>1.0</td>\n",
       "      <td>273.0</td>\n",
       "      <td>21.0</td>\n",
       "      <td>396.90</td>\n",
       "      <td>7.88</td>\n",
       "      <td>11.9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>506 rows × 14 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        CRIM    ZN  INDUS  CHAS    NOX     RM   AGE     DIS  RAD    TAX  \\\n",
       "0    0.00632  18.0   2.31   0.0  0.538  6.575  65.2  4.0900  1.0  296.0   \n",
       "1    0.02731   0.0   7.07   0.0  0.469  6.421  78.9  4.9671  2.0  242.0   \n",
       "2    0.02729   0.0   7.07   0.0  0.469  7.185  61.1  4.9671  2.0  242.0   \n",
       "3    0.03237   0.0   2.18   0.0  0.458  6.998  45.8  6.0622  3.0  222.0   \n",
       "4    0.06905   0.0   2.18   0.0  0.458  7.147  54.2  6.0622  3.0  222.0   \n",
       "..       ...   ...    ...   ...    ...    ...   ...     ...  ...    ...   \n",
       "501  0.06263   0.0  11.93   0.0  0.573  6.593  69.1  2.4786  1.0  273.0   \n",
       "502  0.04527   0.0  11.93   0.0  0.573  6.120  76.7  2.2875  1.0  273.0   \n",
       "503  0.06076   0.0  11.93   0.0  0.573  6.976  91.0  2.1675  1.0  273.0   \n",
       "504  0.10959   0.0  11.93   0.0  0.573  6.794  89.3  2.3889  1.0  273.0   \n",
       "505  0.04741   0.0  11.93   0.0  0.573  6.030  80.8  2.5050  1.0  273.0   \n",
       "\n",
       "     PTRATIO       B  LSTAT  Price  \n",
       "0       15.3  396.90   4.98   24.0  \n",
       "1       17.8  396.90   9.14   21.6  \n",
       "2       17.8  392.83   4.03   34.7  \n",
       "3       18.7  394.63   2.94   33.4  \n",
       "4       18.7  396.90   5.33   36.2  \n",
       "..       ...     ...    ...    ...  \n",
       "501     21.0  391.99   9.67   22.4  \n",
       "502     21.0  396.90   9.08   20.6  \n",
       "503     21.0  396.90   5.64   23.9  \n",
       "504     21.0  393.45   6.48   22.0  \n",
       "505     21.0  396.90   7.88   11.9  \n",
       "\n",
       "[506 rows x 14 columns]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "boston_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "各个特征的相关解释：\n",
    "   - CRIM：各城镇的人均犯罪率\n",
    "   - ZN：规划地段超过25,000平方英尺的住宅用地比例\n",
    "   - INDUS：城镇非零售商业用地比例\n",
    "   - CHAS：是否在查尔斯河边(=1是)\n",
    "   - NOX：一氧化氮浓度(/千万分之一)\n",
    "   - RM：每个住宅的平均房间数\n",
    "   - AGE：1940年以前建造的自住房屋的比例\n",
    "   - DIS：到波士顿五个就业中心的加权距离\n",
    "   - RAD：放射状公路的可达性指数\n",
    "   - TAX：全部价值的房产税率(每1万美元)\n",
    "   - PTRATIO：按城镇分配的学生与教师比例\n",
    "   - B：1000(Bk - 0.63)^2其中Bk是每个城镇的黑人比例\n",
    "   - LSTAT：较低地位人口\n",
    "   - Price：房价"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(2) 选择度量模型性能的指标：                      \n",
    "   - MSE均方误差：$\\text{MSE}(y, \\hat{y}) = \\frac{1}{n_\\text{samples}} \\sum_{i=0}^{n_\\text{samples} - 1} (y_i - \\hat{y}_i)^2.$\n",
    "   - MAE平均绝对误差:$\\text{MAE}(y, \\hat{y}) = \\frac{1}{n_{\\text{samples}}} \\sum_{i=0}^{n_{\\text{samples}}-1} \\left| y_i - \\hat{y}_i \\right|$\n",
    "   - $R^2$决定系数：$R^2(y, \\hat{y}) = 1 - \\frac{\\sum_{i=1}^{n} (y_i - \\hat{y}_i)^2}{\\sum_{i=1}^{n} (y_i - \\bar{y})^2}$\n",
    "   - 解释方差得分:$explained\\_{}variance(y, \\hat{y}) = 1 - \\frac{Var\\{ y - \\hat{y}\\}}{Var\\{y\\}}$\n",
    "\n",
    "https://scikit-learn.org/stable/modules/model_evaluation.html#regression-metrics\n",
    "\n",
    "![jupyter](./1.3.png)              \n",
    "在这个案例中，我们使用MSE均方误差为模型的性能度量指标。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(3) 选择具体的模型并进行训练\n",
    "   - **线性回归模型**         \n",
    "   回归这个概念是19世纪80年代由英国统计学家郎西斯.高尔顿在研究父子身高关系提出来的，他发现：在同一族群中，子代的平均身高介于父代的身高以及族群的平均身高之间。具体而言，高个子父亲的儿子的身高有低于其父亲身高的趋势，而矮个子父亲的儿子身高则有高于父亲的身高的趋势。也就是说，子代的身高有向族群平均身高\"平均\"的趋势，这就是统计学上\"回归\"的最初含义。回归分析是一种预测性的建模技术，它研究的是因变量（目标）和自变量（特征）之间的关系。这种技术通常用于预测分析，时间序列模型以及发现变量之间的因果关系。通常使用曲线/线来拟合数据点，目标是使曲线到数据点的距离差异最小。而线性回归就是回归问题中的一种，线性回归假设目标值与特征之间线性相关，即满足一个多元一次方程。通过构建损失函数，来求解损失函数最小时的参数w ：                                                \n",
    "   假设：数据集$D = \\{(x_1,y_1),...,(x_N,y_N) \\}$，$x_i \\in R^p,y_i \\in R,i = 1,2,...,N$，$X = (x_1,x_2,...,x_N)^T,Y=(y_1,y_2,...,y_N)^T$                        \n",
    "   假设X和Y之间存在线性关系，模型的具体形式为$\\hat{y}=f(w) =w^Tx$           \n",
    "   ![jupyter](./1.4.png)      \n",
    "   (a) 最小二乘估计：                 \n",
    "   我们需要衡量真实值$y_i$与线性回归模型的预测值$w^Tx_i$之间的差距，在这里我们和使用二范数的平方和L(w)来描述这种差距，即：                      \n",
    "   $$\n",
    "   L(w) = \\sum\\limits_{i=1}^{N}||w^Tx_i-y_i||_2^2=\\sum\\limits_{i=1}^{N}(w^Tx_i-y_i)^2 = (w^TX^T-Y^T)(w^TX^T-Y^T)^T = w^TX^TXw - 2w^TX^TY+YY^T\\\\\n",
    "   因此，我们需要找到使得L(w)最小时对应的参数w，即：\\\\\n",
    "   \\hat{w} = argmin\\;L(w)\\\\\n",
    "   为了达到求解最小化L(w)问题，我们应用高等数学的知识，使用求导来解决这个问题： \\\\\n",
    "   \\frac{\\partial L(w)}{\\partial w} = 2X^TXw-2X^TY = 0,因此： \\\\\n",
    "   \\hat{w} = (X^TX)^{-1}X^TY\n",
    "   $$                   \n",
    "   (b) 几何解释：                \n",
    "   在线性代数中，我们知道两个向量a和b相互垂直可以得出：$<a,b> = a.b = a^Tb = 0$,而平面X的法向量为Y-Xw，与平面X互相垂直，因此：$X^T(Y-Xw) = 0$，即：$w = (X^TX)^{-1}X^TY$                             \n",
    "   ![jupyter](./1.5.png)               \n",
    "   (c) 概率视角：       \n",
    "   假设噪声$\\epsilon \\backsim N(0,\\sigma^2),y=f(w)+\\epsilon=w^Tx+\\epsilon$，因此：$y|x_i,w ~ N(w^Tx,\\sigma^2)$          \n",
    "   我们使用极大似然估计MLE对参数w进行估计：       \n",
    "   $$\n",
    "   L(w) = log\\;P(Y|X;w) = log\\;\\prod_{i=1}^N P(y_i|x_i;w) = \\sum\\limits_{i=1}^{N} log\\; P(y_i|x_i;w)\\\\\n",
    "    = \\sum\\limits_{i=1}^{N}log(\\frac{1}{\\sqrt{2\\pi \\sigma}}exp(-\\frac{(y_i-w^Tx_i)^2}{2\\sigma^2})) = \\sum\\limits_{i=1}^{N}[log(\\frac{1}{\\sqrt{2\\pi}\\sigma})-\\frac{1}{2\\sigma^2}(y_i-w^Tx_i)^2] \\\\\n",
    "    argmax_w L(w) = argmin_w[l(w) = \\sum\\limits_{i = 1}^{N}(y_i-w^Tx_i)^2]\\\\\n",
    "    因此：线性回归的最小二乘估计<==>噪声\\epsilon\\backsim N(0,\\sigma^2)的极大似然估计\n",
    "   $$\n",
    "   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面，我们使用sklearn的线性回归实例来演示：                   \n",
    "https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html#sklearn.linear_model.LinearRegression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-14T11:47:46.070536Z",
     "start_time": "2021-03-14T11:47:46.055536Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "模型系数： [-1.08011358e-01  4.64204584e-02  2.05586264e-02  2.68673382e+00\n",
      " -1.77666112e+01  3.80986521e+00  6.92224640e-04 -1.47556685e+00\n",
      "  3.06049479e-01 -1.23345939e-02 -9.52747232e-01  9.31168327e-03\n",
      " -5.24758378e-01]\n",
      "模型得分： 0.7406426641094095\n"
     ]
    }
   ],
   "source": [
    "from sklearn import linear_model      # 引入线性回归方法\n",
    "lin_reg = linear_model.LinearRegression()       # 创建线性回归的类\n",
    "lin_reg.fit(X,y)        # 输入特征X和因变量y进行训练\n",
    "print(\"模型系数：\",lin_reg.coef_)             # 输出模型的系数\n",
    "print(\"模型得分：\",lin_reg.score(X,y))    # 输出模型的决定系数R^2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "   - **线性回归的推广**                     \n",
    "   在线性回归中，我们假设因变量与特征之间的关系是线性关系，这样的假设使得模型很简单，但是缺点也是显然的，那就是当数据存在非线性关系时，我们使用线性回归模型进行预测会导致预测性能极其低下，因为模型的形式本身是线性的，无法表达数据中的非线性关系。我们一个很自然的想法就是去推广线性回归模型，使得推广后的模型更能表达非线性的关系。                     \n",
    "   (a) 多项式回归：        \n",
    "   为了体现因变量和特征的非线性关系，一个很自然而然的想法就是将标准的线性回归模型：             \n",
    "   $$\n",
    "   y_i = w_0 + w_1x_i + \\epsilon_i\n",
    "   $$               \n",
    "   换成一个多项式函数：          \n",
    "   $$\n",
    "   y_i = w_0 + w_1x_i + w_2x_i^2 + ...+w_dx_i^d + \\epsilon\n",
    "   $$                        \n",
    "   对于多项式的阶数d不能取过大，一般不大于3或者4，因为d越大，多项式曲线就会越光滑，在X的边界处有异常的波动。（图中的边界处的4阶多项式拟合曲线的置信区间(虚线表示置信区间)明显增大，预测效果的稳定性下降。）   \n",
    "\n",
    "   ![jupyter](./1.6.1.png) ![jupyter](./1.6.2.png)                             \n",
    "   \n",
    "   (b) 广义可加模型(GAM)：         \n",
    "   广义可加模型GAM实际上是线性模型推广至非线性模型的一个框架，在这个框架中，每一个变量都用一个非线性函数来代替，但是模型本身保持整体可加性。GAM模型不仅仅可以用在线性回归的推广，还可以将线性分类模型进行推广。具体的推广形式是：          \n",
    "   标准的线性回归模型：           \n",
    "   $$\n",
    "   y_i = w_0 + w_1x_{i1} +...+w_px_{ip} + \\epsilon_i \n",
    "   $$                 \n",
    "   GAM模型框架：                  \n",
    "   $$\n",
    "   y_i = w_0 + \\sum\\limits_{j=1}^{p}f_{j}(x_{ij}) + \\epsilon_i\n",
    "   $$                  \n",
    "   GAM模型的优点与不足：             \n",
    "      - 优点：简单容易操作，能够很自然地推广线性回归模型至非线性模型，使得模型的预测精度有所上升；由于模型本身是可加的，因此GAM还是能像线性回归模型一样把其他因素控制不变的情况下单独对某个变量进行推断，极大地保留了线性回归的易于推断的性质。\n",
    "      - 缺点：GAM模型会经常忽略一些有意义的交互作用，比如某两个特征共同影响因变量，不过GAM还是能像线性回归一样加入交互项$x^{(i)} \\times x^{(j)}$的形式进行建模；但是GAM模型本质上还是一个可加模型，如果我们能摆脱可加性模型形式，可能还会提升模型预测精度，详情请看后面的算法。\n",
    "     \n",
    "   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(a) 多项式回归实例介绍：                    \n",
    "https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.PolynomialFeatures.html?highlight=poly#sklearn.preprocessing.PolynomialFeatures                              \n",
    "sklearn.preprocessing.PolynomialFeatures(degree=2, *, interaction_only=False, include_bias=True, order='C'):               \n",
    "   - 参数：         \n",
    "   degree：特征转换的阶数。                       \n",
    "   interaction_onlyboolean：是否只包含交互项，默认False 。              \n",
    "   include_bias：是否包含截距项，默认True。           \n",
    "   order：str in {‘C’, ‘F’}, default ‘C’，输出数组的顺序。     "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "原始X为：\n",
      " [[0 1]\n",
      " [2 3]\n",
      " [4 5]]\n",
      "2次转化X：\n",
      " [[ 1.  0.  1.  0.  0.  1.]\n",
      " [ 1.  2.  3.  4.  6.  9.]\n",
      " [ 1.  4.  5. 16. 20. 25.]]\n",
      "2次转化X：\n",
      " [[ 1.  0.  1.  0.]\n",
      " [ 1.  2.  3.  6.]\n",
      " [ 1.  4.  5. 20.]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "X_arr = np.arange(6).reshape(3, 2)\n",
    "print(\"原始X为：\\n\",X_arr)\n",
    "\n",
    "poly = PolynomialFeatures(2)\n",
    "print(\"2次转化X：\\n\",poly.fit_transform(X_arr)) #[x_0,x_1,x_2,x_1x_2,x_1^2,x_2^2] \n",
    "\n",
    "poly = PolynomialFeatures(interaction_only=True)\n",
    "print(\"2次转化X：\\n\",poly.fit_transform(X_arr))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(b) GAM模型实例介绍：          \n",
    "安装pygam：pip install pygam               \n",
    "https://github.com/dswah/pyGAM/blob/master/doc/source/notebooks/quick_start.ipynb                     "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-14T11:53:47.825598Z",
     "start_time": "2021-03-14T11:53:47.286600Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LinearGAM                                                                                                 \n",
      "=============================================== ==========================================================\n",
      "Distribution:                        NormalDist Effective DoF:                                    103.2423\n",
      "Link Function:                     IdentityLink Log Likelihood:                                 -1589.7653\n",
      "Number of Samples:                          506 AIC:                                             3388.0152\n",
      "                                                AICc:                                            3442.7649\n",
      "                                                GCV:                                               13.7683\n",
      "                                                Scale:                                              8.8269\n",
      "                                                Pseudo R-Squared:                                   0.9168\n",
      "==========================================================================================================\n",
      "Feature Function                  Lambda               Rank         EDoF         P > x        Sig. Code   \n",
      "================================= ==================== ============ ============ ============ ============\n",
      "s(0)                              [0.6]                20           11.2         2.20e-11     ***         \n",
      "s(1)                              [0.6]                20           12.7         8.15e-02     .           \n",
      "s(2)                              [0.6]                20           13.3         2.59e-03     **          \n",
      "s(3)                              [0.6]                20           3.5          2.76e-01                 \n",
      "s(4)                              [0.6]                20           11.4         1.11e-16     ***         \n",
      "s(5)                              [0.6]                20           10.2         1.11e-16     ***         \n",
      "s(6)                              [0.6]                20           10.4         8.22e-01                 \n",
      "s(7)                              [0.6]                20           8.6          4.44e-16     ***         \n",
      "s(8)                              [0.6]                20           3.6          5.96e-03     **          \n",
      "s(9)                              [0.6]                20           3.6          1.33e-09     ***         \n",
      "s(10)                             [0.6]                20           1.7          3.26e-03     **          \n",
      "s(11)                             [0.6]                20           6.4          6.25e-02     .           \n",
      "s(12)                             [0.6]                20           6.6          1.11e-16     ***         \n",
      "intercept                                              1            0.0          2.23e-13     ***         \n",
      "==========================================================================================================\n",
      "Significance codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1\n",
      "\n",
      "WARNING: Fitting splines and a linear function to a feature introduces a model identifiability problem\n",
      "         which can cause p-values to appear significant when they are not.\n",
      "\n",
      "WARNING: p-values calculated in this manner behave correctly for un-penalized models or models with\n",
      "         known smoothing parameters, but when smoothing parameters have been estimated, the p-values\n",
      "         are typically lower than they should be, meaning that the tests reject the null too readily.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-19-bb049c74476f>:3: UserWarning: KNOWN BUG: p-values computed in this summary are likely much smaller than they should be. \n",
      " \n",
      "Please do not make inferences based on these values! \n",
      "\n",
      "Collaborate on a solution, and stay up to date at: \n",
      "github.com/dswah/pyGAM/issues/163 \n",
      "\n",
      "  gam.summary()\n"
     ]
    }
   ],
   "source": [
    "from pygam import LinearGAM\n",
    "gam = LinearGAM().fit(boston_data[boston.feature_names], y)\n",
    "gam.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "   - **回归树：**                                   \n",
    "   基于树的回归方法主要是依据分层和分割的方式将特征空间划分为一系列简单的区域。对某个给定的待预测的自变量，用他所属区域中训练集的平均数或者众数对其进行预测。由于划分特征空间的分裂规则可以用树的形式进行概括，因此这类方法称为决策树方法。决策树由结点(node)和有向边(diredcted edge)组成。结点有两种类型：内部结点(internal node)和叶结点(leaf node)。内部结点表示一个特征或属性，叶结点表示一个类别或者某个值。区域$R_1,R_2$等称为叶节点，将特征空间分开的点为内部节点。                           \n",
    "   ![jupyter](./1.7.png)                     \n",
    "   建立回归树的过程大致可以分为以下两步：          \n",
    "      - 将自变量的特征空间(即$x^{(1)},x^{(2)},x^{(3)},...,x^{(p)}$)的可能取值构成的集合分割成J个互不重叠的区域$R_1,R_2,...,R_j$。        \n",
    "      - 对落入区域$R_j$的每个观测值作相同的预测，预测值等于$R_j$上训练集的因变量的简单算术平均。              \n",
    "   具体来说，就是：             \n",
    "      a.  选择最优切分特征j以及该特征上的最优点s：                \n",
    "      遍历特征j以及固定j后遍历切分点s，选择使得下式最小的(j,s)  $min_{j,s}[min_{c_1}\\sum\\limits_{x_i\\in R_1(j,s)}(y_i-c_1)^2 + min_{c_2}\\sum\\limits_{x_i\\in R_2(j,s)}(y_i-c_2)^2 ]$                           \n",
    "       b. 按照(j,s)分裂特征空间：$R_1(j,s) = \\{x|x^{j} \\le s \\}和R_2(j,s) = \\{x|x^{j} > s \\},\\hat{c}_m = \\frac{1}{N_m}\\sum\\limits_{x \\in R_m(j,s)}y_i,\\;m=1,2$                           \n",
    "       c. 继续调用步骤1，2直到满足停止条件，就是每个区域的样本数小于等于5。        \n",
    "       d. 将特征空间划分为J个不同的区域，生成回归树：$f(x) = \\sum\\limits_{m=1}^{J}\\hat{c}_mI(x \\in R_m)$                \n",
    "       如以下生成的关于运动员在棒球大联盟数据的回归树：             \n",
    "       ![jupyter](./1.8.png)                   \n",
    "    回归树与线性模型的比较：              \n",
    "    线性模型的模型形式与树模型的模型形式有着本质的区别，具体而言，线性回归对模型形式做了如下假定：$f(x) = w_0 + \\sum\\limits_{j=1}^{p}w_jx^{(j)}$，而回归树则是$f(x) = \\sum\\limits_{m=1}^{J}\\hat{c}_mI(x \\in R_m)$。那问题来了，哪种模型更优呢？这个要视具体情况而言，如果特征变量与因变量的关系能很好的用线性关系来表达，那么线性回归通常有着不错的预测效果，拟合效果则优于不能揭示线性结构的回归树。反之，如果特征变量与因变量的关系呈现高度复杂的非线性，那么树方法比传统方法更优。                     \n",
    "    ![jupyter](./1.9.1.png)                        \n",
    "    树模型的优缺点：                 \n",
    "    - 树模型的解释性强，在解释性方面可能比线性回归还要方便。\n",
    "    - 树模型更接近人的决策方式。\n",
    "    - 树模型可以用图来表示，非专业人士也可以轻松解读。\n",
    "    - 树模型可以直接做定性的特征而不需要像线性回归一样哑元化。\n",
    "    - 树模型能很好处理缺失值和异常值，对异常值不敏感，但是这个对线性模型来说却是致命的。\n",
    "    - 树模型的预测准确性一般无法达到其他回归模型的水平，但是改进的方法很多。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "sklearn使用回归树的实例：                \n",
    "https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeRegressor.html?highlight=tree#sklearn.tree.DecisionTreeRegressor                         \n",
    "sklearn.tree.DecisionTreeRegressor(*, criterion='mse', splitter='best', max_depth=None, min_samples_split=2, min_samples_leaf=1, min_weight_fraction_leaf=0.0, max_features=None, random_state=None, max_leaf_nodes=None, min_impurity_decrease=0.0, min_impurity_split=None, presort='deprecated', ccp_alpha=0.0）                                                 \n",
    "   - 参数：(列举几个重要的，常用的，详情请看上面的官网)                 \n",
    "   criterion：{“ mse”，“ friedman_mse”，“ mae”}，默认=“ mse”。衡量分割标准的函数 。                      \n",
    "   splitter：{“best”, “random”}, default=”best”。分割方式。                    \n",
    "   max_depth：树的最大深度。               \n",
    "   min_samples_split：拆分内部节点所需的最少样本数，默认是2。                      \n",
    "   min_samples_leaf：在叶节点处需要的最小样本数。默认是1。                    \n",
    "   min_weight_fraction_leaf：在所有叶节点处（所有输入样本）的权重总和中的最小加权分数。如果未提供sample_weight，则样本的权重相等。默认是0。                      "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-14T11:55:05.005502Z",
     "start_time": "2021-03-14T11:55:04.961503Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9376307599929274"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.tree import DecisionTreeRegressor    \n",
    "reg_tree = DecisionTreeRegressor(criterion = \"mse\",min_samples_leaf = 5)\n",
    "reg_tree.fit(X,y)\n",
    "reg_tree.score(X,y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "   - 支持向量机回归(SVR)                                        \n",
    "   在介绍支持向量回归SVR之前，我们先来了解下约束优化的相关知识：          \n",
    "      - 约束优化问题(P)：                   \n",
    "      $$\n",
    "      min f(x)  \\\\\n",
    "      s.t.\\;\\;\\;g_i(x) \\le 0,\\; i=1,2,...,m\\\\\n",
    "      \\;\\;\\;\\;\\; h_j(x) = 0,\\; j=1,2,...,l\n",
    "      $$                         \n",
    "   我们假设$x^*$为满足以上条件的局部最优解，$p^* = f(x^*)$，我们的目的就是要找到$x^*$与$p^*$，满足不等式和等式约束的x集合成为可行域，记作S。\n",
    "   \n",
    "    - KKT条件(最优解的一阶必要条件)          \n",
    "    因为KKT条件是最优化的相关内容，在本次开源学习中并不是重点，因此在这里我用一个更加简单的例子说明KKT条件，严格的证明请参见凸优化相关书籍。                       \n",
    "    在这个例子中，我们考虑：($x^*$为我们的最优解)               \n",
    "    $$\n",
    "    minf(x)\\\\\n",
    "    s.t.\\;g_1(x) \\le 0,\\;x \\in R^n\\\\\n",
    "    \\;\\;\\;g_2(x) \\le 0\\\\\n",
    "    \\;\\;\\;g_3(x) \\le 0\n",
    "    $$\n",
    "    ![jupyter](./1.11.png)\n",
    "      \n",
    "      我们可以看到：$-\\nabla f(x^*)$可以由$\\nabla g_1(x^*)$与$\\nabla g_2(x^*)$线性表出，因此有：$-\\nabla f(x^*) = \\lambda_1 \\nabla g_1(x^*) + \\lambda_2 \\nabla g_2(x^*)$，其中$\\lambda_1,\\lambda_2 \\ge 0$，即：                          \n",
    "      $$\n",
    "      \\nabla f(x^*) + \\lambda_1 \\nabla g_1(x^*) + \\lambda_2 \\nabla g_2(x^*) = 0,\\;\\;\\;其中\\lambda_1,\\lambda_2 \\ge 0\n",
    "      $$                       \n",
    "      我们把没有起作用的约束$g_3(x)$也放到式子里面去，目的也就是为了书写方便，即要求：                  \n",
    "      $$\n",
    "      \\nabla f(x^*) + \\lambda_1 \\nabla g_1(x^*) + \\lambda_2 \\nabla g_2(x^*) + \\lambda_3 \\nabla g_3(x^*)= 0,\\;\\;\\;其中\\lambda_1,\\lambda_2 \\ge 0,\\lambda_3 = 0\n",
    "      $$                      \n",
    "      由于点$x^*$位于方程$g_1(x)=0$与$g_2(x)=0$上，因此：$\\lambda_1 g_1(x^*)  = 0,\\lambda_2  g_2(x^*) = 0 , \\lambda_3  g_3(x^*)= 0$                        \n",
    "      \n",
    "      因此，KKT条件就是：假设$x^*$为最优化问题(P)的局部最优解，且$x^*$ 在某个适当的条件下 ,有：                             \n",
    "      $$\n",
    "     \\nabla f(x^*) + \\sum\\limits_{i=1}^{m}\\lambda_i \\nabla g(x^*) + \\sum\\limits_{j=1}^{l}\\mu_j \\nabla h_j(x^*) = 0(对偶条件)\\\\     \n",
    "      \\lambda_i \\ge 0,\\;i = 1,2,...,m(对偶条件)\\\\\n",
    "      g_i(x^*) \\le 0(原问题条件)\\\\\n",
    "      h_j(x^*) = 0(原问题条件)\\\\\n",
    "      \\lambda_i g(x^*) = 0(互补松弛定理)\n",
    "      $$                              \n",
    " - 对偶理论：               \n",
    "   为什么要引入对偶问题呢？是因为原问题与对偶问题就像是一个问题两个角度去看，如利润最大与成本最低等。有时侯原问题上难以解决，但是在对偶问题上就会变得很简单。再者，任何一个原问题在变成对偶问题后都会变成一个凸优化的问题，这点我们后面会有介绍。下面我们来引入对偶问题：                           \n",
    "   首先，我们的原问题(P)是：\n",
    "      $$\n",
    "      min f(x)  \\\\\n",
    "      s.t.\\;\\;\\;g_i(x) \\le 0,\\; i=1,2,...,m\\\\\n",
    "      \\;\\;\\;\\;\\; h_j(x) = 0,\\; j=1,2,...,l\n",
    "      $$                             \n",
    "      引入拉格朗日函数：$L(x,\\lambda,\\mu) = f(x) + \\sum\\limits_{i=1}^{m}\\lambda_i g_i(x) + \\sum\\limits_{j=1}^{l}\\mu_j h_j(x)$                        \n",
    "      拉格朗日对偶函数：                                  \n",
    "      $$\n",
    "      d(\\lambda,\\mu)  =  min_{x\\in X}\\{ f(x) + \\sum\\limits_{i=1}^{m}\\lambda_i g_i(x) + \\sum\\limits_{j=1}^{l}\\mu_j h_j(x)\\} ,其中X为满足条件的x变量\\\\\n",
    "      \\le min_{x\\in S}\\{ f(x) + \\sum\\limits_{i=1}^{m}\\lambda_i g_i(x) + \\sum\\limits_{j=1}^{l}\\mu_j h_j(x) \\},由于g_i(x) \\le 0,h_j(x) = 0,\\lambda_i \\ge 0 ,其中S为可行域\\\\\n",
    "      \\le min_{x\\in S}\\{f(x) \\}\n",
    "      $$                                   \n",
    "      因此：拉格朗日对偶函数$d(\\lambda,\\mu)$是原问题最优解的函数值$p^*$的下界，即每个不同的$\\lambda$与$\\mu$确定的$d(\\lambda,\\mu)$都是$p^*$的下界，但是我们希望下界越大越好，因为越大就更能接近真实的$p^*$。因此：                               \n",
    "      拉格朗日对偶问题(D)转化为：                 \n",
    "      $$\n",
    "      max_{\\lambda,\\mu}d(\\lambda,\\mu)\\\\\n",
    "      s.t. \\lambda_i \\ge 0,i = 1,2,...,m\\\\\n",
    "      也就是：\\\\\n",
    "      max_{\\lambda \\ge 0,\\mu}\\;min_{x \\in S} L(x,\\lambda,\\mu)\n",
    "      $$                   \n",
    "      我们可以观察到，对偶问题是关于$\\lambda$和$\\mu$的线性函数，因此对偶问题是一个凸优化问题，凸优化问题在最优化理论较为简单。\n",
    "      弱对偶定理：对偶问题(D)的最优解$D^*$一定小于原问题最优解$P^*$，这点在刚刚的讨论得到了充分的证明，一定成立。                   \n",
    "      强对偶定理：对偶问题(D)的最优解$D^*$在一定的条件下等于原问题最优解$P^*$，条件非常多样化且不是唯一的，也就是说这是个开放性的问题，在这里我给出一个最简单的条件，即：$f(x)$与$g_i(x)$为凸函数，$h_j(x)$为线性函数，X是凸集，$x^*$满足KKT条件，那么$D^* = P^*$。 \n",
    "      \n",
    "   \n",
    "      "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "   - 支持向量回归SVR                    \n",
    "   在介绍完了相关的优化知识以后，我们开始正式学习支持向量回归SVR。                    \n",
    "   ![jupyter](./1.10.png)                         \n",
    "   在线性回归的理论中，每个样本点都要计算平方损失，但是SVR却是不一样的。SVR认为：落在$f(x)$的$\\epsilon$邻域空间中的样本点不需要计算损失，这些都是预测正确的，其余的落在$\\epsilon$邻域空间以外的样本才需要计算损失，因此：              \n",
    "   ![jupyter](./1.12.png)                                      \n",
    "   $$\n",
    "   min_{w,b,\\xi_i,\\hat{\\xi}_i} \\frac{1}{2}||w||^2 +C \\sum\\limits_{i=1}^{N}(\\xi_i,\\hat{\\xi}_i)\\\\\n",
    "   s.t.\\;\\;\\; f(x_i) - y_i \\le \\epsilon + \\xi_i\\\\\n",
    "   \\;\\;\\;\\;\\;y_i - f(x_i) \\le  \\epsilon +\\hat{\\xi}_i\\\\\n",
    "   \\;\\;\\;\\;\\; \\xi_i,\\hat{\\xi}_i \\le 0,i = 1,2,...,N\n",
    "   $$                                 \n",
    "   引入拉格朗日函数：                  \n",
    "   $$\n",
    "   \\begin{array}{l}\n",
    "L(w, b, \\alpha, \\hat{\\alpha}, \\xi, \\xi, \\mu, \\hat{\\mu}) \\\\\n",
    "\\quad=\\frac{1}{2}\\|w\\|^{2}+C \\sum_{i=1}^{N}\\left(\\xi_{i}+\\widehat{\\xi}_{i}\\right)-\\sum_{i=1}^{N} \\xi_{i} \\mu_{i}-\\sum_{i=1}^{N} \\widehat{\\xi}_{i} \\widehat{\\mu}_{i} \\\\\n",
    "\\quad+\\sum_{i=1}^{N} \\alpha_{i}\\left(f\\left(x_{i}\\right)-y_{i}-\\epsilon-\\xi_{i}\\right)+\\sum_{i=1}^{N} \\widehat{\\alpha}_{i}\\left(y_{i}-f\\left(x_{i}\\right)-\\epsilon-\\widehat{\\xi}_{i}\\right)\n",
    "\\end{array}\n",
    "   $$                       \n",
    "   再令$L(w, b, \\alpha, \\hat{\\alpha}, \\xi, \\xi, \\mu, \\hat{\\mu})$对$w,b,\\xi,\\hat{\\xi}$求偏导等于0，得： $w=\\sum_{i=1}^{N}\\left(\\widehat{\\alpha}_{i}-\\alpha_{i}\\right) x_{i}$。                             \n",
    "   上述过程中需满足KKT条件，即要求：                 \n",
    "   $$\n",
    "   \\left\\{\\begin{array}{c}\n",
    "\\alpha_{i}\\left(f\\left(x_{i}\\right)-y_{i}-\\epsilon-\\xi_{i}\\right)=0 \\\\\n",
    "\\hat{\\alpha_{i}}\\left(y_{i}-f\\left(x_{i}\\right)-\\epsilon-\\hat{\\xi}_{i}\\right)=0 \\\\\n",
    "\\alpha_{i} \\widehat{\\alpha}_{i}=0, \\xi_{i} \\hat{\\xi}_{i}=0 \\\\\n",
    "\\left(C-\\alpha_{i}\\right) \\xi_{i}=0,\\left(C-\\widehat{\\alpha}_{i}\\right) \\hat{\\xi}_{i}=0\n",
    "\\end{array}\\right.\n",
    "   $$                         \n",
    "   SVR的解形如:$f(x)=\\sum_{i=1}^{N}\\left(\\widehat{\\alpha}_{i}-\\alpha_{i}\\right) x_{i}^{T} x+b$                                                  \n",
    "   \n",
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "sklearn中使用SVR实例：                        \n",
    "sklearn.svm.SVR(*, kernel='rbf', degree=3, gamma='scale', coef0=0.0, tol=0.001, C=1.0, epsilon=0.1, shrinking=True, cache_size=200, verbose=False, max_iter=-1)            \n",
    "https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVR.html?highlight=svr#sklearn.svm.SVR                        \n",
    "   - 参数：                   \n",
    "   kernel：核函数，{‘linear’, ‘poly’, ‘rbf’, ‘sigmoid’, ‘precomputed’}, 默认=’rbf’。(后面会详细介绍)              \n",
    "   degree：多项式核函数的阶数。默认 = 3。                          \n",
    "   C：正则化参数，默认=1.0。(后面会详细介绍)                            \n",
    "   epsilon：SVR模型允许的不计算误差的邻域大小。默认0.1。              \n",
    "   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-14T11:56:00.141392Z",
     "start_time": "2021-03-14T11:56:00.070391Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7024525421955277"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.svm import SVR\n",
    "from sklearn.preprocessing import StandardScaler     # 标准化数据\n",
    "from sklearn.pipeline import make_pipeline   # 使用管道，把预处理和模型形成一个流程\n",
    "\n",
    "reg_svr = make_pipeline(StandardScaler(), SVR(C=1.0, epsilon=0.2))\n",
    "reg_svr.fit(X, y)\n",
    "reg_svr.score(X,y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(4) 优化基础模型                   \n",
    "在刚刚的回归问题的基本算法中，我们使用数据集去估计模型的参数，如线性回归模型中的参数w，那么这个数据集我们称为训练数据集，简称训练集。我们在回归问题中使用训练集估计模型的参数的原则一般都是使得我们的损失函数在训练集达到最小值，其实在实际问题中我们是可以让损失函数在训练集最小化为0，如：在线性回归中，我加入非常多的高次项，使得我们模型在训练集的每一个数据点都恰好位于曲线上，那这时候模型在训练集的损失值也就是误差为0。                \n",
    "![jupyter](./1.13.png)                            \n",
    "既然能做到这件事，是不是代表我们的建模完事大吉呢？换句话说我们的模型可以预测任意情况呢？答案是显然否定的。我们建立机器学习的目的并不是为了在已有的数据集，也就是训练集上效果表现非常优异，我们希望建立的机器学习模型在未知且情况复杂的测试数据上表现优异，我们称这样的未出现在训练集的未知数据集成为测试数据集，简称测试集。我们希望模型在测试集上表现优异！因为假如我们根据股票市场前六个月的数据拟合一个预测模型，我们的目的不是为了预测以前这六个月越准越好，而是预测明天乃至未来的股价变化。                                  \n",
    "   - (a) 训练均方误差与测试均方误差：                   \n",
    "   在回归中，我们最常用的评价指标为均方误差，即：$MSE = \\frac{1}{N}\\sum\\limits_{i=1}^{N}(y_i -\\hat{ f}(x_i))^2$，其中$\\hat{ f}(x_i)$是样本$x_i$应用建立的模型$\\hat{f}$预测的结果。如果我们所用的数据是训练集上的数据，那么这个误差为训练均方误差，如果我们使用测试集的数据计算的均方误差，我们称为测试均方误差。一般而言，我们并不关心模型在训练集上的训练均方误差，我们关心的是模型面对未知的样本集，即测试集上的测试误差，我们的目标是使得我们建立的模型在测试集上的测试误差最小。那我们如何选择一个测试误差最小的模型呢？这是个棘手的问题，因为在模型建立阶段，我们是不能得到测试数据的，比如：我们在模型未上线之前是不能拿到未知且真实的测试数据来验证我们的模型的。在这种情况下，为了简便起见，一些观点认为通过训练误差最小化来选择模型也是可行的。这种观点表面看上去是可行的，但是存在一个致命的缺点，那就是：一个模型的训练均方误差最小时，不能保证测试均方误差同时也很小。对于这种想法构造的模型，一般在训练误差达到最小时，测试均方误差一般很大！如图：                       \n",
    "   ![jupyter](./1.14.png)                       \n",
    "   ![jupyter](./1.15.png)                    \n",
    "   可以看到：当我们的模型的训练均方误差达到很小时，测试均方误差反而很大，但是我们寻找的最优的模型是测试均方误差达到最小时对应的模型，因此基于训练均方误差达到最小选择模型本质上是行不同的。正如上右图所示：模型在训练误差很小，但是测试均方误差很大时，我们称这种情况叫模型的**过拟合**。                            \n",
    "   - (b) 偏差-方差的权衡：                         \n",
    "   从上图的测试均方误差曲线可以看到：测试均方误差曲线呈现U型曲线，这表明了在测试误差曲线中有两种力量在互相博弈。可以证明：           \n",
    "   $$\n",
    "   E\\left(y_{0}-\\hat{f}\\left(x_{0}\\right)\\right)^{2}=\\operatorname{Var}\\left(\\hat{f}\\left(x_{0}\\right)\\right)+\\left[\\operatorname{Bias}\\left(\\hat{f}\\left(x_{0}\\right)\\right)\\right]^{2}+\\operatorname{Var}(\\varepsilon)\n",
    "   $$                  \n",
    "   也就是说，我们的测试均方误差的期望值可以分解为$\\hat{f}(x_0)$的方差、$\\hat{f}(x_0)$的偏差平方和误差项$\\epsilon$的方差。为了使得模型的测试均方误差达到最小值，也就是同时最小化偏差的平方和方差。由于我们知道偏差平方和方差本身是非负的，因此测试均方误差的期望不可能会低于误差的方差，因此我们称$\\operatorname{Var}(\\varepsilon)$为建模任务的难度，这个量在我们的任务确定后是无法改变的，也叫做不可约误差。那么模型的方差和偏差的平方和究竟是什么呢？所谓模型的方差就是：用不同的数据集去估计$f$时，估计函数的改变量。举个例子：我们想要建立一个线性回归模型，可以通过输入中国人身高去预测我们的体重。但是显然我们没有办法把全中国13亿人做一次人口普查，拿到13亿人的身高体重去建立模型。我们能做的就是从13亿中抽1000个样本进行建模，我们对这个抽样的过程重复100遍，就会得到100个1000人的样本集。我们使用线性回归模型估计参数就能得到100个线性回归模型。由于样本抽取具有随机性，我们得到的100个模型不可能参数完全一样，那么这100个模型之间的差异就叫做方差。显然，我们希望得到一个稳定的模型，也就是在不同的样本集估计的模型都不会相差太大，即要求f的方差越小越好。**一般来说，模型的复杂度越高，f的方差就会越大。** 如加入二次项的模型的方差比线性回归模型的方差要大。                              \n",
    "   ![jupyter](./1.16.png)                                \n",
    "   另一方面，模型的偏差是指：为了选择一个简单的模型去估计真实函数所带入的误差。假如真实的数据X与Y的关系是二次关系，但是我们选择了线性模型进行建模，那由于模型的复杂度引起的这种误差我们称为偏差，它的构成时复杂的。偏差度量了学习算法的期望预测与真实结果的偏离程度，即刻画了学习算法本身的拟合能力。偏差度量的是单个模型的学习能力，而方差度量的是同一个模型在不同数据集上的稳定性。“偏差-方差分解”说明：泛化性能是由学习算法的能力、数据的充分性以及学习任务本身的难度所共同决定的。给定学习任务，为了取得好的泛化性能，则需使偏差较小，即能够充分拟合数据，并且使方差较小，即使得数据扰动产生的影响小。                          \n",
    "   ![jupyter](./1.17.png)                    \n",
    "   一般而言，增加模型的复杂度，会增加模型的方差，但是会减少模型的偏差，我们要找到一个方差--偏差的权衡，使得测试均方误差最。                        \n",
    "   ![jupyter](./1.18.png)                              \n",
    "   (c) 特征提取：              \n",
    "   在前面的讨论中，我们已经明确一个目标，就是：我们要选择一个测试误差达到最小的模型。但是实际上我们很难对实际的测试误差做精确的计算，因此我们要对测试误差进行估计，估计的方式有两种：训练误差修正与交叉验证。                         \n",
    "      - 训练误差修正：      \n",
    "      前面的讨论我们已经知道，模型越复杂，训练误差越小，测试误差先减后增。因此，我们先构造一个特征较多的模型使其过拟合，此时训练误差很小而测试误差很大，那这时我们加入关于特征个数的惩罚。因此，当我们的训练误差随着特征个数的增加而减少时，惩罚项因为特征数量的增加而增大，抑制了训练误差随着特征个数的增加而无休止地减小。具体的数学量如下：             \n",
    "      $C_p = \\frac{1}{N}(RSS  +  2d\\hat{\\sigma}^2)$，其中d为模型特征个数，$RSS = \\sum\\limits_{i=1}^{N}(y_i-\\hat{f}(x_i))^2$，$\\hat{\\sigma}^2$为模型预测误差的方差的估计值，即残差的方差。                       \n",
    "      AIC赤池信息量准则：$AIC = \\frac{1}{d\\hat{\\sigma}^2}(RSS  +  2d\\hat{\\sigma}^2)$                              \n",
    "      BIC贝叶斯信息量准则：$BIC = \\frac{1}{n}(RSS + log(n)d\\hat{\\sigma}^2)$                     \n",
    "      - 交叉验证：                   \n",
    "      前面讨论的对训练误差修正得到测试误差的估计是间接方法，这种方法的桥梁是训练误差，而交叉验证则是对测试误差的直接估计。交叉验证比训练误差修正的优势在于：能够给出测试误差的一个直接估计。在这里只介绍K折交叉验证：我们把训练样本分成K等分，然后用K-1个样本集当做训练集，剩下的一份样本集为验证集去估计由K-1个样本集得到的模型的精度，这个过程重复K次取平均值得到测试误差的一个估计$CV_{(K)} = \\frac{1}{K}\\sum\\limits_{i=1}^{K}MSE_i$。5折交叉验证如下图：（蓝色的是训练集，黄色的是验证集）                     \n",
    "      ![jupyter](./1.19.png)                   \n",
    "   \n",
    "   在测试误差能够被合理的估计出来以后，我们做特征选择的目标就是：从p个特征中选择m个特征，使得对应的模型的测试误差的估计最小。对应的方法有：\n",
    "      - 最优子集选择：                  \n",
    "      (i) 记不含任何特征的模型为$M_0$，计算这个$M_0$的测试误差。                                              \n",
    "      (ii) 在$M_0$基础上增加一个变量，计算p个模型的RSS，选择RSS最小的模型记作$M_1$，并计算该模型$M_1$的测试误差。                                          \n",
    "      (iii) 再增加变量，计算p-1个模型的RSS，并选择RSS最小的模型记作$M_2$，并计算该模型$M_2$的测试误差。                                                \n",
    "      (iv) 重复以上过程知道拟合的模型有p个特征为止，并选择p+1个模型$\\{M_0,M_1,...,M_p \\}$中测试误差最小的模型作为最优模型。                           \n",
    "      - 向前逐步选择：               \n",
    "      最优子集选择虽然在原理上很直观，但是随着数据特征维度p的增加，子集的数量为$2^p$，计算效率非常低下且需要的计算内存也很高，在大数据的背景下显然不适用。因此，我们需要把最优子集选择的运算效率提高，因此向前逐步选择算法的过程如下：                            \n",
    "      (i) 记不含任何特征的模型为$M_0$，计算这个$M_0$的测试误差。                    \n",
    "      (ii) 在$M_0$基础上增加一个变量，计算p个模型的RSS，选择RSS最小的模型记作$M_1$，并计算该模型$M_1$的测试误差。                  \n",
    "      (iii) 在最小的RSS模型下继续增加一个变量，选择RSS最小的模型记作$M_2$，并计算该模型$M_2$的测试误差。                  \n",
    "      (iv) 以此类推，重复以上过程知道拟合的模型有p个特征为止，并选择p+1个模型$\\{M_0,M_1,...,M_p \\}$中测试误差最小的模型作为最优模型。                      \n",
    "  \n",
    "  (d) 压缩估计(正则化)：                         \n",
    "  除了刚刚讨论的直接对特征自身进行选择以外，我们还可以对回归的系数进行约束或者加罚的技巧对p个特征的模型进行拟合，显著降低模型方差，这样也会提高模型的拟合效果。具体来说，就是将回归系数往零的方向压缩，这也就是为什么叫压缩估计的原因了。                          \n",
    "     - 岭回归(L2正则化的例子)：                 \n",
    "     在线性回归中，我们的损失函数为$J(w) = \\sum\\limits_{i=1}^{N}(y_i-w_0-\\sum\\limits_{j=1}^{p}w_jx_{ij})^2$，我们在线性回归的损失函数的基础上添加对系数的约束或者惩罚，即：                           \n",
    "     $$\n",
    "     J(w) = \\sum\\limits_{i=1}^{N}(y_i-w_0-\\sum\\limits_{j=1}^{p}w_jx_{ij})^2 + \\lambda\\sum\\limits_{j=1}^{p}w_j^2,\\;\\;其中，\\lambda \\ge 0\\\\\n",
    "     \\hat{w} = (X^TX + \\lambda I)^{-1}X^TY\n",
    "     $$                        \n",
    "   调节参数$\\lambda$的大小是影响压缩估计的关键，$\\lambda$越大，惩罚的力度越大，系数则越趋近于0，反之，选择合适的$\\lambda$对模型精度来说十分重要。岭回归通过牺牲线性回归的无偏性降低方差，有可能使得模型整体的测试误差较小，提高模型的泛化能力。                                 \n",
    "     - Lasso回归(L1正则化的例子)：             \n",
    "     岭回归的一个很显著的特点是：将模型的系数往零的方向压缩，但是岭回归的系数只能趋于0但无法等于0，换句话说，就是无法做特征选择。能否使用压缩估计的思想做到像特征最优子集选择那样提取出重要的特征呢？答案是肯定的！我们只需要对岭回归的优化函数做小小的调整就行了，我们使用系数向量的L1范数替换岭回归中的L2范数：                     \n",
    "     $$\n",
    "     J(w) = \\sum\\limits_{i=1}^{N}(y_i-w_0-\\sum\\limits_{j=1}^{p}w_jx_{ij})^2 + \\lambda\\sum\\limits_{j=1}^{p}|w_j|,\\;\\;其中，\\lambda \\ge 0\n",
    "     $$                  \n",
    "   为什么Losso能做到特征选择而岭回归却不能呢个做到呢？(如图：左边为Lasso，右边为岭回归)                \n",
    "   ![jupyter](./1.20.png)                       \n",
    "   椭圆形曲线为RSS等高线，菱形和圆形区域分别代表了L1和L2约束，Lasso回归和岭回归都是在约束下的回归，因此最优的参数为椭圆形曲线与菱形和圆形区域相切的点。但是Lasso回归的约束在每个坐标轴上都有拐角，因此当RSS曲线与坐标轴相交时恰好回归系数中的某一个为0，这样就实现了特征提取。反观岭回归的约束是一个圆域，没有尖点，因此与RSS曲线相交的地方一般不会出现在坐标轴上，因此无法让某个特征的系数为0，因此无法做到特征提取。                                \n",
    "   \n",
    "   (e) 降维：                \n",
    "   到目前为止，我们所讨论的方法对方差的控制有两种方式：一种是使用原始变量的子集，另一种是将变量系数压缩至零。但是这些方法都是基于原始特征$x_1,...,x_p$得到的，现在我们探讨一类新的方法：将原始的特征空间投影到一个低维的空间实现变量的数量变少，如：将二维的平面投影至一维空间。机器学习领域中所谓的降维就是指采用某种映射方法，将原高维空间中的数据点映射到低维度的空间中。降维的本质是学习一个映射函数 f : x->y，其中x是原始数据点的表达，目前最多使用向量表达形式。 y是数据点映射后的低维向量表达，通常y的维度小于x的维度（当然提高维度也是可以的）。f可能是显式的或隐式的、线性的或非线性的。目前大部分降维算法处理向量表达的数据，也有一些降维算法处理高阶张量表达的数据。之所以使用降维后的数据表示是因为在原始的高维空间中，包含有冗余信息以及噪音信息，在实际应用例如图像识别中造成了误差，降低了准确率；而通过降维,我们希望减少 冗余信息 所造成的误差,提高识别（或其他应用）的精度。又或者希望通过降维算法来寻找数据内部的本质结构特征。在很多算法中，降维算法成为了数据预处理的一部分，如PCA。事实上，有一些算法如果没有降维预处理，其实是很难得到很好的效果的。 (摘自：rosenor1博客)                                \n",
    "   主成分分析(PCA)：                     \n",
    "   主成分分析的思想：通过**最大投影方差** 将原始空间进行重构，即由特征相关重构为无关，即落在某个方向上的点(投影)的方差最大。在进行下一步推导之前，我们先把样本均值和样本协方差矩阵推广至矩阵形式：                     \n",
    "   样本均值Mean:$\\bar{x} = \\frac{1}{N}\\sum\\limits_{i=1}^{N}x_i =  \\frac{1}{N}X^T1_N,\\;\\;\\;其中1_N = (1,1,...,1)_{N}^T$                         \n",
    "   样本协方差矩阵$S^2 = \\frac{1}{N}\\sum\\limits_{i=1}^{N}(x_i-\\bar{x})(x_i-\\bar{x})^T = \\frac{1}{N}X^THX,\\;\\;\\;其中，H = I_N - \\frac{1}{N}1_N1_N^T$                             \n",
    "   最大投影方差的步骤：                \n",
    "   (i) 中心化：$x_i - \\bar{x}$                       \n",
    "   (ii) 计算每个点$x_1,...,x_N$至$\\vec{u}_1$方向上的投影：$(x_i-\\bar{x})\\vec{u}_1,\\;\\;\\;||\\vec{u}_1|| = 1$                             \n",
    "   (iii) 计算投影方差：$J = \\frac{1}{N}\\sum\\limits_{i=1}^{N}[(x_i-\\bar{x})^T\\vec{u}_1]^2,\\;\\;\\;||\\vec{u}_1|| = 1$                           \n",
    "   (iv) 最大化投影方差求$\\vec{u}_1$：                       \n",
    "   $$\n",
    "   \\bar{u}_1 = argmax_{u_1}\\;\\;\\frac{1}{N}\\sum\\limits_{i=1}^{N}[(x_i-\\bar{x})^T\\vec{u}_1]^2 \\\\\n",
    "   \\;\\;\\;s.t. \\vec{u}_1^T\\vec{u}_1 = 1 (\\vec{u}_1往后不带向量符号)\n",
    "   $$                         \n",
    "   得到：               \n",
    "   $$\n",
    "   J = \\frac{1}{N}\\sum\\limits_{i=1}^{N}[(x_i-\\bar{x})^T\\vec{u}_1]^2  = \\frac{1}{N}\\sum\\limits_{i=1}^{N}[u_1^T(x_i-\\bar{x})(x_i-\\bar{x})^Tu_1]\\\\\n",
    "   \\; = u_1^T[\\frac{1}{N}\\sum\\limits_{i=1}^{N}(x_i-\\bar{x})(x_i - \\bar{x})^T]u_1 = u_1^TS^2u_1\n",
    "   $$                    \n",
    "   即：           \n",
    "   $$\n",
    "   \\hat{u}_1 = argmax_{u_1}u_1^TS^2u_1,\\;\\;\\;s.t.u_1^Tu_1 = 1\\\\\n",
    "   L(u_1,\\lambda) = u_1^TS^2u_1 + \\lambda (1-u_1^Tu_1)\\\\\n",
    "   \\frac{\\partial L}{\\partial u_1} = 2S^2u_1-2\\lambda u_1 = 0\\\\\n",
    "   即：S^2u_1 = \\lambda u_1\n",
    "   $$                        \n",
    "   可以看到：$\\lambda$为$S^2$的特征值，$u_1$为$S^2$的特征向量。因此我们只需要对中心化后的协方差矩阵进行特征值分解，得到的特征向量即为投影方向。如果需要进行降维，那么只需要取p的前M个特征向量即可。\n",
    "   \n",
    "    \n",
    "   \n",
    "      \n",
    "      \n",
    "   \n",
    "\n",
    "\n",
    "     \n",
    "   \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "特征提取的实例：向前逐步回归                      \n",
    "案例来源：https://blog.csdn.net/weixin_44835596/article/details/89763300                            \n",
    "根据AIC准则定义向前逐步回归进行变量筛选"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-14T12:14:19.500400Z",
     "start_time": "2021-03-14T12:14:19.485401Z"
    }
   },
   "outputs": [],
   "source": [
    "#定义向前逐步回归函数\n",
    "def forward_select(data,target):\n",
    "    variate=set(data.columns)  #将字段名转换成字典类型\n",
    "    variate.remove(target)  #去掉因变量的字段名\n",
    "    selected=[]\n",
    "    current_score,best_new_score=float('inf'),float('inf')  #目前的分数和最好分数初始值都为无穷大（因为AIC越小越好）\n",
    "    #循环筛选变量\n",
    "    while variate:\n",
    "        aic_with_variate=[]\n",
    "        for candidate in variate:  #逐个遍历自变量\n",
    "            formula=\"{}~{}\".format(target,\"+\".join(selected+[candidate]))  #将自变量名连接起来\n",
    "            aic=ols(formula=formula,data=data).fit().aic  #利用ols训练模型得出aic值\n",
    "            aic_with_variate.append((aic,candidate))  #将第每一次的aic值放进空列表\n",
    "        aic_with_variate.sort(reverse=True)  #降序排序aic值\n",
    "        best_new_score,best_candidate=aic_with_variate.pop()  #最好的aic值等于删除列表的最后一个值，以及最好的自变量等于列表最后一个自变量\n",
    "        if current_score>best_new_score:  #如果目前的aic值大于最好的aic值\n",
    "            variate.remove(best_candidate)  #移除加进来的变量名，即第二次循环时，不考虑此自变量了\n",
    "            selected.append(best_candidate)  #将此自变量作为加进模型中的自变量\n",
    "            current_score=best_new_score  #最新的分数等于最好的分数\n",
    "            print(\"aic is {},continuing!\".format(current_score))  #输出最小的aic值\n",
    "        else:\n",
    "            print(\"Selection over!\")\n",
    "            break\n",
    "    formula=\"{}~{}\".format(target,\"+\".join(selected))  #最终的模型式子\n",
    "    print(\"Final formula is: {}\".format(formula))\n",
    "    model=ols(formula=formula,data=data).fit()\n",
    "    return(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-14T12:14:20.656426Z",
     "start_time": "2021-03-14T12:14:20.112430Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "aic is 3286.974956900157,continuing!\n",
      "aic is 3171.5423142992013,continuing!\n",
      "aic is 3114.0972674193326,continuing!\n",
      "aic is 3097.359044862759,continuing!\n",
      "aic is 3069.438633167217,continuing!\n",
      "aic is 3057.9390497191152,continuing!\n",
      "aic is 3048.438382711162,continuing!\n",
      "aic is 3042.274993098419,continuing!\n",
      "aic is 3040.154562175143,continuing!\n",
      "aic is 3032.0687017003256,continuing!\n",
      "aic is 3021.726387825062,continuing!\n",
      "Selection over!\n",
      "Final formula is: Price~LSTAT+RM+PTRATIO+DIS+NOX+CHAS+B+ZN+CRIM+RAD+TAX\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<statsmodels.regression.linear_model.RegressionResultsWrapper at 0x1e60292de50>"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import statsmodels.api as sm #最小二乘\n",
    "from statsmodels.formula.api import ols #加载ols模型\n",
    "forward_select(data=boston_data,target=\"Price\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-14T12:18:39.748590Z",
     "start_time": "2021-03-14T12:18:39.706590Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>OLS Regression Results</caption>\n",
       "<tr>\n",
       "  <th>Dep. Variable:</th>          <td>Price</td>      <th>  R-squared:         </th> <td>   0.741</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model:</th>                   <td>OLS</td>       <th>  Adj. R-squared:    </th> <td>   0.735</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Method:</th>             <td>Least Squares</td>  <th>  F-statistic:       </th> <td>   128.2</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Date:</th>             <td>Sun, 14 Mar 2021</td> <th>  Prob (F-statistic):</th> <td>5.54e-137</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Time:</th>                 <td>20:18:39</td>     <th>  Log-Likelihood:    </th> <td> -1498.9</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>No. Observations:</th>      <td>   506</td>      <th>  AIC:               </th> <td>   3022.</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Df Residuals:</th>          <td>   494</td>      <th>  BIC:               </th> <td>   3072.</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Df Model:</th>              <td>    11</td>      <th>                     </th>     <td> </td>    \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Covariance Type:</th>      <td>nonrobust</td>    <th>                     </th>     <td> </td>    \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "      <td></td>         <th>coef</th>     <th>std err</th>      <th>t</th>      <th>P>|t|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Intercept</th> <td>   36.3411</td> <td>    5.067</td> <td>    7.171</td> <td> 0.000</td> <td>   26.385</td> <td>   46.298</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>LSTAT</th>     <td>   -0.5226</td> <td>    0.047</td> <td>  -11.019</td> <td> 0.000</td> <td>   -0.616</td> <td>   -0.429</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>RM</th>        <td>    3.8016</td> <td>    0.406</td> <td>    9.356</td> <td> 0.000</td> <td>    3.003</td> <td>    4.600</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>PTRATIO</th>   <td>   -0.9465</td> <td>    0.129</td> <td>   -7.334</td> <td> 0.000</td> <td>   -1.200</td> <td>   -0.693</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>DIS</th>       <td>   -1.4927</td> <td>    0.186</td> <td>   -8.037</td> <td> 0.000</td> <td>   -1.858</td> <td>   -1.128</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>NOX</th>       <td>  -17.3760</td> <td>    3.535</td> <td>   -4.915</td> <td> 0.000</td> <td>  -24.322</td> <td>  -10.430</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>CHAS</th>      <td>    2.7187</td> <td>    0.854</td> <td>    3.183</td> <td> 0.002</td> <td>    1.040</td> <td>    4.397</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>B</th>         <td>    0.0093</td> <td>    0.003</td> <td>    3.475</td> <td> 0.001</td> <td>    0.004</td> <td>    0.015</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>ZN</th>        <td>    0.0458</td> <td>    0.014</td> <td>    3.390</td> <td> 0.001</td> <td>    0.019</td> <td>    0.072</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>CRIM</th>      <td>   -0.1084</td> <td>    0.033</td> <td>   -3.307</td> <td> 0.001</td> <td>   -0.173</td> <td>   -0.044</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>RAD</th>       <td>    0.2996</td> <td>    0.063</td> <td>    4.726</td> <td> 0.000</td> <td>    0.175</td> <td>    0.424</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>TAX</th>       <td>   -0.0118</td> <td>    0.003</td> <td>   -3.493</td> <td> 0.001</td> <td>   -0.018</td> <td>   -0.005</td>\n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "  <th>Omnibus:</th>       <td>178.430</td> <th>  Durbin-Watson:     </th> <td>   1.078</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Prob(Omnibus):</th> <td> 0.000</td>  <th>  Jarque-Bera (JB):  </th> <td> 787.785</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Skew:</th>          <td> 1.523</td>  <th>  Prob(JB):          </th> <td>8.60e-172</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Kurtosis:</th>      <td> 8.300</td>  <th>  Cond. No.          </th> <td>1.47e+04</td> \n",
       "</tr>\n",
       "</table><br/><br/>Notes:<br/>[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.<br/>[2] The condition number is large, 1.47e+04. This might indicate that there are<br/>strong multicollinearity or other numerical problems."
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary.Summary'>\n",
       "\"\"\"\n",
       "                            OLS Regression Results                            \n",
       "==============================================================================\n",
       "Dep. Variable:                  Price   R-squared:                       0.741\n",
       "Model:                            OLS   Adj. R-squared:                  0.735\n",
       "Method:                 Least Squares   F-statistic:                     128.2\n",
       "Date:                Sun, 14 Mar 2021   Prob (F-statistic):          5.54e-137\n",
       "Time:                        20:18:39   Log-Likelihood:                -1498.9\n",
       "No. Observations:                 506   AIC:                             3022.\n",
       "Df Residuals:                     494   BIC:                             3072.\n",
       "Df Model:                          11                                         \n",
       "Covariance Type:            nonrobust                                         \n",
       "==============================================================================\n",
       "                 coef    std err          t      P>|t|      [0.025      0.975]\n",
       "------------------------------------------------------------------------------\n",
       "Intercept     36.3411      5.067      7.171      0.000      26.385      46.298\n",
       "LSTAT         -0.5226      0.047    -11.019      0.000      -0.616      -0.429\n",
       "RM             3.8016      0.406      9.356      0.000       3.003       4.600\n",
       "PTRATIO       -0.9465      0.129     -7.334      0.000      -1.200      -0.693\n",
       "DIS           -1.4927      0.186     -8.037      0.000      -1.858      -1.128\n",
       "NOX          -17.3760      3.535     -4.915      0.000     -24.322     -10.430\n",
       "CHAS           2.7187      0.854      3.183      0.002       1.040       4.397\n",
       "B              0.0093      0.003      3.475      0.001       0.004       0.015\n",
       "ZN             0.0458      0.014      3.390      0.001       0.019       0.072\n",
       "CRIM          -0.1084      0.033     -3.307      0.001      -0.173      -0.044\n",
       "RAD            0.2996      0.063      4.726      0.000       0.175       0.424\n",
       "TAX           -0.0118      0.003     -3.493      0.001      -0.018      -0.005\n",
       "==============================================================================\n",
       "Omnibus:                      178.430   Durbin-Watson:                   1.078\n",
       "Prob(Omnibus):                  0.000   Jarque-Bera (JB):              787.785\n",
       "Skew:                           1.523   Prob(JB):                    8.60e-172\n",
       "Kurtosis:                       8.300   Cond. No.                     1.47e+04\n",
       "==============================================================================\n",
       "\n",
       "Notes:\n",
       "[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.\n",
       "[2] The condition number is large, 1.47e+04. This might indicate that there are\n",
       "strong multicollinearity or other numerical problems.\n",
       "\"\"\""
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lm=ols(\"Price~LSTAT+RM+PTRATIO+DIS+NOX+CHAS+B+ZN+CRIM+RAD+TAX\",data=boston_data).fit()\n",
    "lm.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "岭回归实例分享：                   \n",
    "sklearn.linear_model.ridge_regression(X, y, alpha, *, sample_weight=None, solver='auto', max_iter=None, tol=0.001, verbose=0, random_state=None, return_n_iter=False, return_intercept=False, check_input=True)                          \n",
    "https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.ridge_regression.html?highlight=rid#sklearn.linear_model.ridge_regression                      \n",
    "   - 参数：                     \n",
    "   alpha：较大的值表示更强的正则化。浮点数                   \n",
    "   sample_weight：样本权重，默认无。                      \n",
    "   solver：求解方法，{‘auto’, ‘svd’, ‘cholesky’, ‘lsqr’, ‘sparse_cg’, ‘sag’, ‘saga’}, 默认=’auto’。“ svd”使用X的奇异值分解来计算Ridge系数。'cholesky'使用标准的scipy.linalg.solve函数通过dot（XT，X）的Cholesky分解获得封闭形式的解。'sparse_cg'使用scipy.sparse.linalg.cg中的共轭梯度求解器。作为一种迭代算法，对于大规模数据（可能设置tol和max_iter），此求解器比“ Cholesky”更合适。 lsqr”使用专用的正则化最小二乘例程scipy.sparse.linalg.lsqr。它是最快的，并且使用迭代过程。“ sag”使用随机平均梯度下降，“ saga”使用其改进的无偏版本SAGA。两种方法都使用迭代过程，并且当n_samples和n_features都很大时，通常比其他求解器更快。请注意，只有在比例大致相同的要素上才能确保“ sag”和“ saga”快速收敛。您可以使用sklearn.preprocessing中的缩放器对数据进行预处理。最后五个求解器均支持密集和稀疏数据。但是，当fit_intercept为True时，仅'sag'和'sparse_cg'支持稀疏输入。                             \n",
    "   \n",
    "   \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.739957023371629"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn import linear_model\n",
    "reg_rid = linear_model.Ridge(alpha=.5)\n",
    "reg_rid.fit(X,y)\n",
    "reg_rid.score(X,y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lasso实例分享：                  \n",
    "class sklearn.linear_model.Lasso(alpha=1.0, *, fit_intercept=True, normalize=False, precompute=False, copy_X=True, max_iter=1000, tol=0.0001, warm_start=False, positive=False, random_state=None, selection='cyclic')                           \n",
    "https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Lasso.html?highlight=lasso#sklearn.linear_model.Lasso                                \n",
    "   - 参数：                \n",
    "   alpha：正则化强度，1.0代表标准最小二乘。                \n",
    "   fit_intercept：是否计算模型截距。默认true。                         \n",
    "   normalize：是否标准化，默认false。                   \n",
    "   positive：是否强制系数为正，默认false。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7140164719858566"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn import linear_model\n",
    "reg_lasso = linear_model.Lasso(alpha = 0.5)\n",
    "reg_lasso.fit(X,y)\n",
    "reg_lasso.score(X,y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(5) 对模型超参数进行调优(调参)：                             \n",
    "在刚刚的讨论中，我们似乎对模型的优化都是对模型算法本身的改进，比如：岭回归对线性回归的优化在于在线性回归的损失函数中加入L2正则化项从而牺牲无偏性降低方差。但是，大家是否想过这样的问题：在L2正则化中参数$\\lambda$应该选择多少？是0.01、0.1、还是1？到目前为止，我们只能凭经验或者瞎猜，能不能找到一种方法找到最优的参数$\\lambda$？事实上，找到最佳参数的问题本质上属于最优化的内容，因为从一个参数集合中找到最佳的值本身就是最优化的任务之一，我们脑海中浮现出来的算法无非就是：梯度下降法、牛顿法等无约束优化算法或者约束优化算法，但是在具体验证这个想法是否可行之前，我们必须先认识两个最本质概念的区别。                                     \n",
    "   - 参数与超参数：                            \n",
    "   我们很自然的问题就是岭回归中的参数$\\lambda$和参数w之间有什么不一样？事实上，参数w是我们通过设定某一个具体的$\\lambda$后使用类似于最小二乘法、梯度下降法等方式优化出来的，我们总是设定了$\\lambda$是多少后才优化出来的参数w。因此，类似于参数w一样，使用最小二乘法或者梯度下降法等最优化算法优化出来的数我们称为参数，类似于$\\lambda$一样，我们无法使用最小二乘法或者梯度下降法等最优化算法优化出来的数我们称为超参数。                                       \n",
    "   模型参数是模型内部的配置变量，其值可以根据数据进行估计。                       \n",
    "      - 进行预测时需要参数。                     \n",
    "      - 它参数定义了可使用的模型。                        \n",
    "      - 参数是从数据估计或获悉的。                       \n",
    "      - 参数通常不由编程者手动设置。                     \n",
    "      - 参数通常被保存为学习模型的一部分。                      \n",
    "      - 参数是机器学习算法的关键，它们通常由过去的训练数据中总结得出 。                          \n",
    "   模型超参数是模型外部的配置，其值无法从数据中估计。\n",
    "      - 超参数通常用于帮助估计模型参数。\n",
    "      - 超参数通常由人工指定。\n",
    "      - 超参数通常可以使用启发式设置。\n",
    "      - 超参数经常被调整为给定的预测建模问题。                            \n",
    "   我们前面(4)部分的优化都是基于模型本身的具体形式的优化，那本次(5)调整的内容是超参数，也就是取不同的超参数的值对于模型的性能有不同的影响。                             \n",
    "   - 网格搜索GridSearchCV()：                \n",
    "   网格搜索：https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html?highlight=gridsearchcv#sklearn.model_selection.GridSearchCV                         \n",
    "   网格搜索结合管道：https://scikit-learn.org/stable/auto_examples/compose/plot_compare_reduction.html?highlight=gridsearchcv                              \n",
    "   网格搜索的思想非常简单，比如你有2个超参数需要去选择，那你就把所有的超参数选择列出来分别做排列组合。举个例子：$\\lambda = 0.01,0.1,1.0$和$\\alpha = 0.01,0.1,1.0$,你可以做一个排列组合，即：{[0.01,0.01],[0.01,0.1],[0.01,1],[0.1,0.01],[0.1,0.1],[0.1,1.0],[1,0.01],[1,0.1],[1,1]}  ，然后针对每组超参数分别建立一个模型，然后选择测试误差最小的那组超参数。换句话说，我们需要从超参数空间中寻找最优的超参数，很像一个网格中找到一个最优的节点，因此叫网格搜索。                         \n",
    "   - 随机搜索 RandomizedSearchCV() ：               \n",
    "   https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.RandomizedSearchCV.html?highlight=randomizedsearchcv#sklearn.model_selection.RandomizedSearchCV                           \n",
    "   网格搜索相当于暴力地从参数空间中每个都尝试一遍，然后选择最优的那组参数，这样的方法显然是不够高效的，因为随着参数类别个数的增加，需要尝试的次数呈指数级增长。有没有一种更加高效的调优方式呢？那就是使用随机搜索的方式，这种方式不仅仅高校，而且实验证明，随机搜索法结果比稀疏化网格法稍好(有时候也会极差，需要权衡)。参数的随机搜索中的每个参数都是从可能的参数值的分布中采样的。与网格搜索相比，这有两个主要优点：        \n",
    "      - 可以独立于参数数量和可能的值来选择计算成本。                 \n",
    "      - 添加不影响性能的参数不会降低效率。                           "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面我们使用SVR的例子，结合管道来进行调优："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-14T12:19:45.684778Z",
     "start_time": "2021-03-14T12:19:45.444778Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CV accuracy: 0.187 +/- 0.649\n"
     ]
    }
   ],
   "source": [
    "# 我们先来对未调参的SVR进行评价： \n",
    "from sklearn.svm import SVR     # 引入SVR类\n",
    "from sklearn.pipeline import make_pipeline   # 引入管道简化学习流程\n",
    "from sklearn.preprocessing import StandardScaler # 由于SVR基于距离计算，引入对数据进行标准化的类\n",
    "from sklearn.model_selection import GridSearchCV  # 引入网格搜索调优\n",
    "from sklearn.model_selection import cross_val_score # 引入K折交叉验证\n",
    "from sklearn import datasets\n",
    "\n",
    "\n",
    "boston = datasets.load_boston()     # 返回一个类似于字典的类\n",
    "X = boston.data\n",
    "y = boston.target\n",
    "features = boston.feature_names\n",
    "pipe_SVR = make_pipeline(StandardScaler(),\n",
    "                         SVR())\n",
    "score1 = cross_val_score(estimator=pipe_SVR,\n",
    "                         X = X,\n",
    "                         y = y,\n",
    "                         scoring = 'r2',\n",
    "                          cv = 10)       # 10折交叉验证\n",
    "print(\"CV accuracy: %.3f +/- %.3f\" % ((np.mean(score1)),np.std(score1)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-14T12:21:38.043502Z",
     "start_time": "2021-03-14T12:19:54.033501Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "网格搜索最优得分： 0.608130307081714\n",
      "网格搜索最优参数组合：\n",
      " {'svr__C': 1000.0, 'svr__gamma': 0.001, 'svr__kernel': 'rbf'}\n"
     ]
    }
   ],
   "source": [
    "# 下面我们使用网格搜索来对SVR调参：\n",
    "from sklearn.pipeline import Pipeline\n",
    "pipe_svr = Pipeline([(\"StandardScaler\",StandardScaler()),\n",
    "                     (\"svr\",SVR())])\n",
    "param_range = [0.0001,0.001,0.01,0.1,1.0,10.0,100.0,1000.0]\n",
    "param_grid = [{\"svr__C\":param_range,\"svr__kernel\":[\"linear\"]},  # 注意__是指两个下划线，一个下划线会报错的\n",
    "                {\"svr__C\":param_range,\"svr__gamma\":param_range,\"svr__kernel\":[\"rbf\"]}]\n",
    "gs = GridSearchCV(estimator=pipe_svr,\n",
    "                     param_grid = param_grid,\n",
    "                     scoring = 'r2',\n",
    "                      cv = 10)       # 10折交叉验证\n",
    "gs = gs.fit(X,y)\n",
    "print(\"网格搜索最优得分：\",gs.best_score_)\n",
    "print(\"网格搜索最优参数组合：\\n\",gs.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-14T12:21:41.823502Z",
     "start_time": "2021-03-14T12:21:38.045502Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "随机搜索最优得分： 0.2989252638654093\n",
      "随机搜索最优参数组合：\n",
      " {'svr__C': 4.7194788472928675, 'svr__gamma': 3.185382674032835, 'svr__kernel': 'linear'}\n"
     ]
    }
   ],
   "source": [
    "# 下面我们使用随机搜索来对SVR调参：\n",
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "from scipy.stats import uniform  # 引入均匀分布设置参数\n",
    "pipe_svr = Pipeline([(\"StandardScaler\",StandardScaler()),\n",
    "                     (\"svr\",SVR())])\n",
    "\n",
    "distributions = dict(svr__C=uniform(loc=1.0, scale=4),    # 构建连续参数的分布\n",
    "                     svr__kernel=[\"linear\",\"rbf\"],        # 离散参数的集合\n",
    "                    svr__gamma=uniform(loc=0, scale=4))\n",
    "\n",
    "rs = RandomizedSearchCV(estimator=pipe_svr,\n",
    "                         param_distributions = distributions,\n",
    "                         scoring = 'r2',\n",
    "                          cv = 10)       # 10折交叉验证\n",
    "rs = rs.fit(X,y)\n",
    "print(\"随机搜索最优得分：\",rs.best_score_)\n",
    "print(\"随机搜索最优参数组合：\\n\",rs.best_params_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "经过我们不懈的努力，从收集数据集并选择合适的特征、选择度量模型性能的指标、选择具体的模型并进行训练以优化模型到评估模型的性能并调参，我们认识到了如何使用sklearn构建简单回归模型。在本章的最后，我们会给出一个具体的案例，整合回归的内容。下面我们来看看机器学习另外一类大问题：分类。与回归一样，分类问题在机器学习的地位非常重要，甚至有的地方用的比回归问题还要多，因此分类问题是十分重要的！"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2 使用sklearn构建完整的分类项目                   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(1) 收集数据集并选择合适的特征：在数据集上我们使用我们比较熟悉的IRIS鸢尾花数据集。\n",
    "  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-14T12:35:18.397590Z",
     "start_time": "2021-03-14T12:35:18.379590Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sepal length (cm)</th>\n",
       "      <th>sepal width (cm)</th>\n",
       "      <th>petal length (cm)</th>\n",
       "      <th>petal width (cm)</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5.1</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4.9</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.7</td>\n",
       "      <td>3.2</td>\n",
       "      <td>1.3</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4.6</td>\n",
       "      <td>3.1</td>\n",
       "      <td>1.5</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5.0</td>\n",
       "      <td>3.6</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   sepal length (cm)  sepal width (cm)  petal length (cm)  petal width (cm)  \\\n",
       "0                5.1               3.5                1.4               0.2   \n",
       "1                4.9               3.0                1.4               0.2   \n",
       "2                4.7               3.2                1.3               0.2   \n",
       "3                4.6               3.1                1.5               0.2   \n",
       "4                5.0               3.6                1.4               0.2   \n",
       "\n",
       "   target  \n",
       "0       0  \n",
       "1       0  \n",
       "2       0  \n",
       "3       0  \n",
       "4       0  "
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn import datasets\n",
    "iris = datasets.load_iris()\n",
    "X = iris.data\n",
    "y = iris.target\n",
    "feature = iris.feature_names\n",
    "data = pd.DataFrame(X,columns=feature)\n",
    "data['target'] = y\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "各个特征的相关解释：\n",
    "   - sepal length (cm)：花萼长度(厘米)\n",
    "   - sepal width (cm)：花萼宽度(厘米)\n",
    "   - petal length (cm)：花瓣长度(厘米)\n",
    "   - petal width (cm)：花瓣宽度(厘米)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(2) 选择度量模型性能的指标：                                    \n",
    "度量分类模型的指标和回归的指标有很大的差异，首先是因为分类问题本身的因变量是离散变量，因此像定义回归的指标那样，单单衡量预测值和因变量的相似度可能行不通。其次，在分类任务中，我们对于每个类别犯错的代价不尽相同，例如：我们将癌症患者错误预测为无癌症和无癌症患者错误预测为癌症患者，在医院和个人的代价都是不同的，前者会使得患者无法得到及时的救治而耽搁了最佳治疗时间甚至付出生命的代价，而后者只需要在后续的治疗过程中继续取证就好了，因此我们很不希望出现前者，当我们发生了前者这样的错误的时候会认为建立的模型是很差的。为了解决这些问题，我们必须将各种情况分开讨论，然后给出评价指标。             \n",
    "   - 真阳性TP：预测值和真实值都为正例；                        \n",
    "   - 真阴性TN：预测值与真实值都为正例；                     \n",
    "   - 假阳性FP：预测值为正，实际值为负；\n",
    "   - 假阴性FN：预测值为负，实际值为正；                      \n",
    "   ![jupyter](./1.22.png)                                       \n",
    "分类模型的指标：                    \n",
    "   - 准确率：分类正确的样本数占总样本的比例，即：$ACC = \\frac{TP+TN}{FP+FN+TP+TN}$.                                \n",
    "   - 精度：预测为正且分类正确的样本占预测值为正的比例，即：$PRE = \\frac{TP}{TP+FP}$.                     \n",
    "   - 召回率：预测为正且分类正确的样本占类别为正的比例，即：$REC =  \\frac{TP}{TP+FN}$.                     \n",
    "   - F1值：综合衡量精度和召回率，即：$F1 = 2\\frac{PRE\\times REC}{PRE + REC}$.                                     \n",
    "   - ROC曲线：以假阳率为横轴，真阳率为纵轴画出来的曲线，曲线下方面积越大越好。                                                          \n",
    "https://scikit-learn.org/stable/modules/model_evaluation.html#classification-metrics                           \n",
    "![jupyter](./1.21.png)                          \n",
    "在本次小案例中，我们使用ROC曲线作为最终评价指标。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(3) 选择具体的模型并进行训练                              \n",
    "   - **逻辑回归logistic regression：**                      \n",
    "   说到分类问题与回归问题的区别，在于回归问题与分类问题需要预测的因变量不一样。在回归问题中，因变量是连续性变量，我们需要预测$E(Y|X)$是一个连续的实数，但是在分类问题中，我们往往是通过已知X的信息预测Y的类别，往往是一个离散集合中的某个元素。如：是否患癌症，图片是猫还是狗等。一个很自然的想法是能否用线性回归去处理分类问题，答案是可以但不好！先来看看线性回归处理分类问题会出现什么弊端，我们仔细来看这个线性回归的例子，${default = \\beta_0 + \\beta_1 Balance + \\beta_2 Income}$，只要输入Balance 和 Income 以及default的数据就能用最小二乘法估计出${\\beta_0,\\beta_1}$,设定预测的default>0.5就是违约反之不违约，感觉很完美的样子，但事实真的是这样吗？假设我们需要用某个人的债务(Balance)和收入(Income)去预测是否会信用卡违约(default)：       \n",
    "      - 我们假设有一个穷人Lisa,他的Balance和Income都很小，那么有可能会导致default的值为负数，那么这个负数代表什么意义呢？显然是没有任何意义的。                \n",
    "      ![jupyter](./1.23.png)                            \n",
    "      - 当我们的分类变量是多类的时候，以0.5为界限划分分类就不可用了，那么我们应该怎么找到一个界限衡量多分类呢？                              \n",
    "   基于以上问题，现在大家是否还觉得线性回归模型作为一个分类模型是否足够优秀呢？其实，为了解决以上的问题（1）我们来想想能不能将线性回归的结果default转化为区间[0:1]上，让default转变成一个违约的概率呢？下面我们来解决这个问题吧。                              \n",
    "   在推导逻辑回归之前，我们先来认识下一组函数，这组函数具有神奇的作用，可以将是实数轴上的数转换为[0:1]区间上的概率。\n",
    "  首先，我们假设我们的线性回归模型为 ${Y=\\beta_0+\\beta_1 X}$，那么这个函数是如何将线性回归的结果转化为概率呢？这个函数就是logistic 函数，具体的形式为   ${p(X) = \\dfrac{e^{\\beta_0 + \\beta_1X}}{1+e^{\\beta_0 + \\beta_1X}}}$，他的函数图像如下图：（左边是线性回归，右边是逻辑函数）                             \n",
    "  ![jupyter](./1.24.png)                                   \n",
    "  因此，我们假设逻辑回归模型为：$p(y = 1|x) = \\frac{1}{1+e^{-w^Tx}}$ .                              \n",
    "  下面我们来具体推导下逻辑回归模型：                          \n",
    "  假设数据Data$\\{(x_i,y_i) \\},\\;\\;i = 1,2,...,N,\\;\\;x_i \\in R^p,y_i \\in \\{0,1 \\}$，设$p_1 = p(y=1|x) = \\sigma(w^T) = \\frac{1}{1+e^{-w^Tx}}$。因为y只可能取0或者1，因此假设数据服从0-1分布，也叫伯努力分布，即：当y=1时，$p(y|x)=p_1$，当y=0时，$p(y|x)=1-p_1$，可以写成$p(y|x) = p_1^y(1-p_1)^{1-y}$，可以带入y=0和y=1进去验证，结果和前面的结论一模一样。                    \n",
    "  我们使用极大似然估计MLE，即：                         \n",
    "  $$\n",
    "  \\hat{w} = argmax_w\\;\\;log\\;P(Y|X) = argmax_x\\;\\;log\\;\\prod_{i=1}^N P(y_i|x_i) = argmax_w \\sum\\limits_{i=1}^{N} log\\;P(y_i|x_i)\\\\\n",
    "  \\;\\;\\; = argmax_w \\sum\\limits_{i=1}^{N}(y_ilog\\;p_1 + (1-y_i)log(1-p_1)) \\\\ \n",
    "  记：L(w) = \\sum\\limits_{i=1}^{N}(y_ilog\\;p_1 + (1-y_i)log(1-p_1))\\\\\n",
    " \\;\\;\\; \\frac{\\partial L}{\\partial w_k} = \\sum\\limits_{i=1}^{N} y_i\\frac{1}{p_1}\\frac{\\partial p_1}{\\partial z}\\frac{\\partial z}{\\partial w_k} + (1-y_i)\\frac{1}{1-p_1}(-\\frac{\\partial p_1}{\\partial z}\\frac{\\partial z}{\\partial w_k})\\\\\n",
    "  \\;\\;\\;=\\sum\\limits_{i=1}^{N}y_i\\frac{1}{\\sigma(z)}(\\sigma(z_i)-\\sigma(z_i)^2)x_i + (1-y_i)\\frac{1}{1-\\sigma(z_i)}[-(\\sigma(z_i)-\\sigma(z_i)^2)x_i]\\\\\n",
    "  \\;\\;\\; =\\sum\\limits_{i=1}^{N}[(y_i-y_i\\sigma(z_i))x_i + (1-y_i)(-\\sigma(z_i))x_i]\\\\\n",
    "  \\;\\;\\; = \\sum\\limits_{i=1}^{N}y_ix_i-\\sigma(z_i)x_i = \\sum\\limits_{i=1}^{N}(y_i-\\sigma(z_i))x_i\n",
    "  $$                 \n",
    "  因此，$\\frac{\\partial L}{\\partial w_k} = \\sum\\limits_{i=1}^{N}(y_i-\\sigma(z_i))x_i$，由于这里涉及的函数不像线性回归一样能简单求出解析解，因此我们使用迭代的优化算法：梯度下降法，即：                       \n",
    "  $w_k^{(t+1)}\\leftarrow w_k^{(t)} - \\eta \\sum\\limits_{i=1}^{N}(y_i-\\sigma(z_i))x_i^{(k)},\\;\\;\\;其中，x_i^{(k)}为第i个样本第k个特征$                                 \n",
    "  如果想了解关于梯度下降法等无约束算法的具体细节，可以参照笔者写的另外两篇知乎博客：                          \n",
    "  最优化理论之无约束优化基本结构及其python应用：https://zhuanlan.zhihu.com/p/163405865                                                   \n",
    "  最优化理论之负梯度方法与Newton型方法：https://zhuanlan.zhihu.com/p/165914126                                              \n",
    "  对于问题(2),我们值得注意的是，逻辑回归在实际中不太用于多分类问题，因为实际效果不是很好，所以我们可以借助其他模型来解决这个问题，那让我们来解决这个遗留下来的问题吧。                                \n",
    " \n",
    "             "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "   - 基于概率的分类模型：                               \n",
    "   (1) 线性判别分析：                                              \n",
    "   线性判别分析是一个比较久远的算法，我将会从两个方向去描述这个算法，因为我觉得每位读者都有自己喜欢的那个理解的方向，分别是基于贝叶斯公式和降维分类的思想。                        \n",
    "      - 基于贝叶斯公式对线性判别分析的理解：                       \n",
    "   在讨论如何解决多分类问题之前，我们先来说说贝叶斯的那些事吧。在概率统计的领域里有一条神奇的公式叫贝叶斯定理，具体的形式是：${P(Y=k|X=x) = \\dfrac{{\\pi}_kf_k(x)}{\\sum\\limits_{l=1}^K{\\pi}_lf_l(x)}}$ ，我们 先不要被公式的符号吓到，我们先来看看符号具体代表什么意思。我们假设观测有${K}$类，${\\pi_k}$为随机选择的观测来自第${k}$类的 __先验概率__，也就是样本里面第${k}$类的样本个数除以总样本的个数：${\\pi_k = \\dfrac{n_k}{n}}$。再来 ${f_k(x) =P(X=x|Y=k)}$，表示第${k}$类观测的X的密度函数，说的直白一点就是在${Y=k}$的样本里${X=x}$的样本个数，即${f_k(x) = P(X=x|Y=k) = \\dfrac{n_{(X=x,Y=k)}}{n_{(Y=k)}}}$，最后，${\\sum\\limits_{l=1}^K{\\pi}_lf_l(x)}=P(X=x)=\\dfrac{n_{(X=x)}}{n}$，也就是样本中${X=x}$的概率。\n",
    "      在讨论贝叶斯定理后，我们回到分类问题，这个定理跟我们的分类问题有什么关联呢？没错，这个公式${P(Y=k|X=x) = \\dfrac{{\\pi}_kf_k(x)}{\\sum\\limits_{l=1}^K{\\pi}_lf_l(x)}}$给出了给定样本条件下，${Y=k}$这个类别下的概率，这给分类问题提供了一条思路，那就是计算这个${P(Y=k|X=x)}$，而且我们的逻辑回归就是这么干的，但是在${P(Y=k|X=x) = \\dfrac{{\\pi}_kf_k(x)}{\\sum\\limits_{l=1}^K{\\pi}_lf_l(x)}}$这个公式中，分母${{\\sum\\limits_{l=1}^K{\\pi}_lf_l(x)} = P(X=x)}$当样本给定的时候是一个与分类${k}$无关的常数,所以我们的问题可以简化为只需要计算分子${{\\pi}_kf_k(x)}$,进而比较哪个类别的概率最大就知道属于哪个类别了，因此我们的分类思路就出来啦，这个思路不同于逻辑回归，逻辑回归需要计算具体的${P(Y=k|X=x)}$概率值，而我们现在的思路是通过贝叶斯定理计算贝叶斯定理的分子，比较分子最大的那个类别为最终类别。                 \n",
    "      在我们推导复杂算法之前，我们先推导下简单的当自变量个数只有一个的模型，即${p=1}$的简单模型。我们记${P(Y=k|X=x) = \\dfrac{{\\pi}_kf_k(x)}{\\sum\\limits_{l=1}^K{\\pi}_lf_l(x)}}$ 的分子为${g_k(x) = {\\pi}_kf_k(x)}$。在这里，我们做个模型假设：假设${f_k(x) }$服从正态分布，即${f_k(x) \\sim N(\\mu,\\sigma_k^2)}$，而且每个${\\sigma_k^2 = \\sigma^2}$，同方差假设。因此${f_k(x) = \\dfrac{1}{\\sqrt{2\\pi}\\sigma_k}e^{-\\dfrac{1}{2\\sigma^2}(x-\\mu_k)^2}}$，最终我们的${g_k(x) = \\pi_k\\dfrac{1}{\\sqrt{2\\pi}\\sigma_k}e^{-\\dfrac{1}{2\\sigma^2}(x-\\mu_k)^2}}$,终于算出来啦。这个式子不是很好计算，我们对${g_k(x)}$取个对数，令${\\delta_k(x) = ln(g_k(x))=ln\\pi_k+\\dfrac{\\mu}{\\sigma^2}x-\\dfrac{\\mu^2}{2\\sigma^2}}$，到这里我们的模型建立模型，我们只需要把未知的${\\mu_k}$与${\\sigma^2}$估计出来就好了。${\\hat{\\mu}_k =\\dfrac{1}{n_k}\\sum\\limits_{i:y_i=k}x_i}$，也就是当${y=k}$这一类中${x}$的平均值；${\\hat{\\sigma}^2 =\\dfrac{1}{n-K}\\sum\\limits_{k=1}^K\\sum\\limits_{i:y_i=k}(x_i-\\hat{\\mu}_k)^2 }$，说白了就是计算每一类的方差，再求平均值。总结下上面的公式就是：                                    \n",
    "${\\begin{cases}\\delta_k(x) = ln(g_k(x))=ln\\pi_k+\\dfrac{\\mu}{\\sigma^2}x-\\dfrac{\\mu^2}{2\\sigma^2}\\\\{\\hat{\\mu}_k =\\dfrac{1}{n_k}\\sum\\limits_{i:y_i=k}x_i}\\\\{\\hat{\\sigma}^2 =\\dfrac{1}{n-K}\\sum\\limits_{k=1}^K\\sum\\limits_{i:y_i=k}(x_i-\\hat{\\mu}_k)^2}\\end{cases}}$                              \n",
    "      至此，我们的模型就建立完成了，我们只需要代入数据求出${\\delta_k(x)}$，哪个${k}$对应的${\\delta_k(x)}$大，就是哪一类。                                   \n",
    "   （下图虚线是线性判别分析的决策边界，正态曲线哪边高样本就是哪一类）                  \n",
    "      ![jupyter](./1.25.png)                            \n",
    "      我们推到出了一个自变量的简单模型，就要泛化为多个自变量的线性判别分析了，即${p>1}$。其实原理一样的，只是将一元正态分布扩展为多元正态分布：\n",
    "      ${f_k(x)=\\dfrac{1}{(2\\pi)^{\\tfrac{p}{2}}|\\Sigma|^\\tfrac{1}{2}}e^{[-\\tfrac{1}{2}(x-\\mu_k)^T\\Sigma^{-1}(x-\\mu_k)]}}$                           \n",
    "      ${\\hat{\\mu_k}=(\\mu_{k1},\\mu_{k2},......,\\mu_{kp})   ,   \\hat{\\Sigma}=\\dfrac{1}{p-1}\\sum\\limits_{j=1}^p(x_j-\\overline{x})(x_j-\\overline{x})^T}$                               \n",
    "      ${\\delta_k(x) = ln(\\pi_kf_k(x))=ln(\\pi_k)-(\\dfrac{p}{2}ln(2\\pi)+\\dfrac{1}{2}ln(|\\Sigma|))-\\dfrac{1}{2}(x-\\mu_k)^T\\Sigma^-1(x-\\mu_k)=x^T\\hat{\\Sigma}\\hat{\\mu}_k-\\dfrac{1}                                                       {2}\\hat{\\mu}_k^T\\hat{\\Sigma}^{-1}\\hat{\\mu}_k+ln\\hat{\\pi}_k}$                            \n",
    "      - 降维分类的思想理解线性判别分析：                   \n",
    "      基于数据进行分类时，一个很自然的想法是：将高维的数据降维至一维，然后使用某个阈值将各个类别分开。下面用图的形式展示：                   \n",
    "      ![jupyter](./1.26.png)                        \n",
    "      图中，数据的维度是二维的，我们的想法是把数据降维至一维，然后用阈值就能分类。这个似乎是一个很好的想法，我们总是希望降维后的数据同一个类别自身内部方差小，不同类别之间的方差要尽可能大。这也是合理的，因为同一个类别的数据应该更加相似，因此方差小；不同类别的数据之间应该很不相似，这样才能更容易对数据进行分类，我们简称为：**类内方差小，类间方差大**，在计算机语言叫“松耦合，高内聚”。在做具体的推导之前，我们对数据的形式和一些基本统计量做一些描述：                            \n",
    "      特征$X = (x_1,x_2,...,x_N)^T$，因变量$Y = (y_1,y_2,...,y_N)^T,\\;\\;其中，y_i \\in \\{+1,-1 \\}$，类别c1的特征$X_{c_1} = \\{x_i|y_i=+1 \\}$，同理，类别c2的特征$X_{c_2} = \\{x_i|y_i=-1 \\}$，属于c1类别的数据个数为$N_1$，属于类别c2的数据个数为$N_2$，其中，$N_1+N_2 = N$。                         \n",
    "      特征X投影在w方向至一维：$z_i = w^Tx_i,\\;\\;||w|| = 1$                            \n",
    "      全样本投影的均值$\\bar{z} = \\frac{1}{N}\\sum\\limits_{i=1}^{N}z_i = \\frac{1}{N}\\sum\\limits_{i=1}^{N}w^Tx_i$                    \n",
    "      全样本投影的协方差$S_z = \\frac{1}{N}\\sum\\limits_{i=1}^{N}(z_i-\\bar{z})(z_i-\\bar{z})^T = \\frac{1}{N}\\sum\\limits_{i=1}^{N}(w^Tx_i-\\bar{z})(w^Tx_i-\\bar{z})^T$                   \n",
    "      c1样本投影的均值$\\bar{z_1} = \\frac{1}{N_1}\\sum\\limits_{i=1}^{N_1}z_i = \\frac{1}{N_1}\\sum\\limits_{i=1}^{N_1}w^Tx_i$                    \n",
    "      c1样本投影的协方差$S_{z_1} = \\frac{1}{N_1}\\sum\\limits_{i=1}^{N_1}(z_i-\\bar{z_1})(z_i-\\bar{z_1})^T = \\frac{1}{N_1}\\sum\\limits_{i=1}^{N_1}(w^Tx_i-\\bar{z_1})(w^Tx_i-\\bar{z_1})^T$                       \n",
    "      c2样本投影的均值 $\\bar{z_2} = \\frac{1}{N_2}\\sum\\limits_{i=1}^{N_2}z_i = \\frac{1}{N_2}\\sum\\limits_{i=1}^{N_2}w^Tx_i$                     \n",
    "      c2样本投影的协方差$S_{z_2} = \\frac{1}{N_2}\\sum\\limits_{i=1}^{N_2}(z_i-\\bar{z_2})(z_i-\\bar{z_2})^T = \\frac{1}{N_2}\\sum\\limits_{i=1}^{N_2}(w^Tx_i-\\bar{z_2})(w^Tx_i-\\bar{z_2})^T$                      \n",
    "      类间差距：$(\\bar{z}_1-\\bar{z}_2)^2$                      \n",
    "      类内方差：$S_1 + S_2$                          \n",
    "      由于线性判别分析的目标是同一类别内方差小，不同类别之间距离大，因此损失函数定义为：   \n",
    "                            \n",
    "   $$\n",
    "      J(w) = \\frac{(\\bar{z}_1-\\bar{z}_2)^2}{s_1+s_2} = \\frac{w^T(\\bar{x}_{c_1}-\\bar{x}_{c_2})(\\bar{x}_{c_1}-\\bar{x}_{c_2})^Tw}{w^T(s_{c_1}+s_{c_2})w}\\\\\n",
    "      \\;\\;\\; \\hat{w} = argmax_w\\;J(w)\n",
    "   $$                             \n",
    "   记：$S_b = (\\bar{x}_{c_1}-\\bar{x}_{c_2})(\\bar{x}_{c_1}-\\bar{x}_{c_2})^T,\\;S_w = (s_{c_1}+s_{c_2})$，因此$J(w) = \\frac{w^TS_bw}{w^TS_ww}$                   \n",
    "   让J(w)对w求导等于0，求出：$w = S_w^{-1}(\\bar{x}_{c_1}-\\bar{x}_{c_2})$                       \n",
    "   (2) 朴素贝叶斯：                                        \n",
    "   在线性判别分析中，我们假设每种分类类别下的特征遵循同一个协方差矩阵，每两个特征之间是存在协方差的，因此在线性判别分析中各种特征是不是独立的。但是，朴素贝叶斯算法对线性判别分析作进一步的模型简化，它将线性判别分析中的协方差矩阵中的协方差全部变成0，只保留各自特征的方差，也就是朴素贝叶斯假设各个特征之间是不相关的。在之前所看到的偏差-方差理论中，我们知道模型的简化可以带来方差的减少但是增加偏差，因此朴素贝叶斯也不例外，它比线性判别分析模型的方差小，偏差大。虽然简化了模型，实际中使用朴素贝叶斯的案例非常多，甚至多于线性判别分析，例如鼎鼎大名的新闻分类，垃圾邮件分类等。\n",
    "   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-14T12:35:23.668587Z",
     "start_time": "2021-03-14T12:35:23.637593Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "e:\\a_projects\\pyprojects\\py_38_project\\venv\\lib\\site-packages\\sklearn\\linear_model\\_logistic.py:763: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  n_iter_i = _check_optimize_result(\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.9733333333333334"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#  逻辑回归\n",
    "'''\n",
    "penalty       {‘l1’, ‘l2’, ‘elasticnet’, ‘none’}, default=’l2’正则化方式\n",
    "dual      bool, default=False   是否使用对偶形式，当n_samples> n_features时，默认dual = False。   \n",
    "C        float, default=1.0      \n",
    "solver       {‘newton-cg’, ‘lbfgs’, ‘liblinear’, ‘sag’, ‘saga’}, default=’lbfgs’     \n",
    "l1_ratio         float, default=None           \n",
    "'''\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "log_iris = LogisticRegression()\n",
    "log_iris.fit(X,y)\n",
    "log_iris.score(X,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-14T12:36:00.016035Z",
     "start_time": "2021-03-14T12:35:59.990036Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.98"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 线性判别分析\n",
    "'''\n",
    "参数：\n",
    "solver:{'svd'，'lsqr'，'eigen'}，默认='svd'\n",
    "solver的使用，可能的值：\n",
    "'svd'：奇异值分解（默认）。不计算协方差矩阵，因此建议将此求解器用于具有大量特征的数据。\n",
    "\n",
    "'lsqr'：最小二乘解，可以与收缩结合使用。\n",
    "\n",
    "'eigen'：特征值分解，可以与收缩结合使用。\n",
    "'''\n",
    "from sklearn.discriminant_analysis import LinearDiscriminantAnalysis\n",
    "lda_iris = LinearDiscriminantAnalysis()\n",
    "lda_iris.fit(X,y)\n",
    "lda_iris.score(X,y)\n",
    "   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-14T12:36:06.451598Z",
     "start_time": "2021-03-14T12:36:06.437598Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.96"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 朴素贝叶斯             \n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "NB_iris = GaussianNB()\n",
    "NB_iris.fit(X, y)\n",
    "NB_iris.score(X,y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "   - 决策树 ：                     \n",
    "   与前面内容所讲的决策树回归大致是一样的，只是在回归问题中，选择分割点的标准是均方误差，但是在分类问题中，由于因变量是类别变量而不是连续变量，因此用均方误差显然不合适。那问题是用什么作为选择分割点的标准呢？我们先来分析具体的问题：                         \n",
    "   在回归树中，对一个给定的观测值，因变量的预测值取它所属的终端结点内训练集的平均因变量。与之相对应，对于分类树来说，给定一个观测值，因变量的预测值为它所属的终端结点内训练集的**最常出现的类**。分类树的构造过程与回归树也很类似，与回归树一样，分类树也是采用递归二叉分裂。但是在分类树中，均方误差无法作为确定分裂节点的准则，一个很自然的替代指标是分类错误率。分类错误率就是：此区域内的训练集中非常见类所占的类别，即：                                   \n",
    "   $$\n",
    "   E = 1-max_k(\\hat{p}_{mk})\n",
    "   $$                       \n",
    "   上式中的$\\hat{p}_{mk}$代表第m个区域的训练集中第k类所占的比例。但是在大量的事实证明：分类错误率在构建决策树时不够敏感，一般在实际中用如下两个指标代替：             \n",
    "   (1) 基尼系数：                   \n",
    "   $$\n",
    "   G = \\sum\\limits_{k=1}^{K} \\hat{p}_{mk}(1-\\hat{p}_{mk})\n",
    "   $$             \n",
    "   在基尼系数的定义中，我们发现这个指标衡量的是K个类别的总方差。不难发现，如果所有的$\\hat{p}_{mk}$的取值都接近0或者1，基尼系数会很小。因此基尼系数被视为衡量结点纯度的指标----如果他的取值小，那就意味着某个节点包含的观测值几乎来自同一个类别。                         \n",
    "   由基尼系数作为指标得到的分类树叫做：CART。                        \n",
    "   (2) 交叉熵：                       \n",
    "   可以替代基尼系数的指标是交叉熵，定义如下：                           \n",
    "   $$\n",
    "   D = -\\sum\\limits_{k=1}^{K} \\hat{p}_{mk}log\\;\\hat{p}_{mk}\n",
    "   $$                     \n",
    "   显然，如果所有的$\\hat{p}_{mk}$都接近于0或者1，那么交叉熵就会接近0。因此，和基尼系数一样，如果第m个结点的纯度越高，则交叉熵越小。事实证明，基尼系数和交叉熵在数值上时很接近的。                   \n",
    "   \n",
    "   ![jupyter](./1.27.png)                                            \n",
    "   决策树分类算法的完整步骤：                          \n",
    "      a.  选择最优切分特征j以及该特征上的最优点s：                \n",
    "      遍历特征j以及固定j后遍历切分点s，选择使得基尼系数或者交叉熵最小的(j,s)                                                   \n",
    "       b. 按照(j,s)分裂特征空间，每个区域内的类别为该区域内样本比例最多的类别。                           \n",
    "       c. 继续调用步骤1，2直到满足停止条件，就是每个区域的样本数小于等于5。        \n",
    "       d. 将特征空间划分为J个不同的区域，生成分类树。                 \n",
    "   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-14T12:40:47.337041Z",
     "start_time": "2021-03-14T12:40:47.324041Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9733333333333334"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 使用决策树算法对iris分类：\n",
    "'''\n",
    "criterion:{“gini”, “entropy”}, default=”gini”\n",
    "max_depth:树的最大深度。\n",
    "min_samples_split:拆分内部节点所需的最少样本数\n",
    "min_samples_leaf :在叶节点处需要的最小样本数。\n",
    "\n",
    "'''\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "tree_iris = DecisionTreeClassifier(min_samples_leaf=5)\n",
    "tree_iris.fit(X,y)\n",
    "tree_iris.score(X,y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "   - 支持向量机SVM：                                   \n",
    "   支持向量机SVM是20世纪90年代在计算机界发展起来的一种分类算法，在许多问题中都被证明有较好的效果，被认为是适应性最广的算法之一。                         \n",
    "   ![jupyter](./1.28.png)                               \n",
    "   支持向量机的基本原理非常简单，如图所视，白色和蓝色的点各为一类，我们的目标是找到一个分割平面将两个类别分开。通常来说，如果数据本身是线性可分的，那么事实上存在无数个这样的超平面。这是因为给定一个分割平面稍微上移下移或旋转这个超平面，只要不接触这些观测点，仍然可以将数据分开。一个很自然的想法就是找到**最大间隔超平面**，即找到一个分割平面距离最近的观测点最远。下面我们来严格推导：                   \n",
    "   我们根据距离超平米那最近的点，只要同时缩放w和b可以得到：$w^Tx_1 + b = 1$与$w^Tx_2+b = -1$，因此：                      \n",
    "   $$\n",
    "  \\begin{array}{l}\n",
    "   w^{T} x_{1}+b=1 \\\\\n",
    "    w^{T} x_{2}+b=-1 \\\\\n",
    "    \\left(w^{T} x_{1}+b\\right)-\\left(w^{T} x_{2}+b\\right)=2 \\\\\n",
    "   w^{T}\\left(x_{1}-x_{2}\\right)=2 \\\\\n",
    "   \\qquad \\begin{array}{l}\n",
    "   w^{T}\\left(x_{1}-x_{2}\\right)=\\|w\\|_{2}\\left\\|x_{1}-x_{2}\\right\\|_{2} \\cos \\theta=2 \\\\\n",
    "   \\left\\|x_{1}-x_{2}\\right\\|_{2} \\cos \\theta=\\frac{2}{\\|w\\|_{2}}\n",
    "   \\end{array} \\\\\n",
    "    \\qquad \\begin{array}{l}\n",
    "   d_{1}=d_{2}=\\frac{\\left\\|x_{1}-x_{2}\\right\\|_{2} \\cos \\theta}{2}=\\frac{\\frac{2}{\\|w\\|_{2}}}{2}=\\frac{1}{\\|w\\|_{2}} \\\\\n",
    "   d_{1}+d_{2}=\\frac{2}{\\|w\\|_{2}}\n",
    "   \\end{array}\n",
    "   \\end{array}\n",
    "   $$                        \n",
    "   由此可知道SVM模型的具体形式：                           \n",
    "   $$\n",
    "  \\begin{aligned}\n",
    "\\min _{w, b} & \\frac{1}{2}\\|w\\|^{2} \\\\\n",
    "\\text { s.t. } & y^{(i)}\\left(w^{T} x^{(i)}+b\\right) \\geq 1, \\quad i=1, \\ldots, n\n",
    "\\end{aligned}\n",
    "   $$                     \n",
    "   可以将约束条件写为: $g_{i}(w)=-y^{(i)}\\left(w^{T} x^{(i)}+b\\right)+1 \\leq 0  $                                                     \n",
    "   可以将优化问题拉格朗日化\n",
    "   $$\n",
    "   \\mathcal{L}(w, b, \\alpha)=\\frac{1}{2}\\|w\\|^{2}-\\sum_{i=1}^{n} \\alpha_{i}\\left[y^{(i)}\\left(w^{T} x^{(i)}+b\\right)-1\\right]\n",
    "   $$                        \n",
    "   因此：                   \n",
    "   $$\n",
    "   \\mathcal{L}(w, b, \\alpha)=\\frac{1}{2}\\|w\\|^{2}-\\sum_{i=1}^{n} \\alpha_{i}\\left[y^{(i)}\\left(w^{T} x^{(i)}+b\\right)-1\\right]\n",
    "   $$                   \n",
    "   欲构造 dual 问题, 首先求拉格朗日化的问题中  $\\mathrm{w} $ 和  $\\mathrm{b} $ 的值, 对 $ \\mathrm{w}$  求梯度, 令梯度为  0,  可求得 w:              \n",
    "   对 b 求梯度, 令梯度为 0, 可得：              \n",
    "   $$\n",
    "   \\frac{\\partial}{\\partial b} \\mathcal{L}(w, b, \\alpha)=\\sum_{i=1}^{n} \\alpha_{i} y^{(i)}=0\n",
    "   $$               \n",
    "\n",
    "   将  $\\mathrm{w}$  带入拉格朗日化的原问题可得                        \n",
    "   $$\n",
    "   \\begin{array}{l}\n",
    "   \\mathcal{L}(w, b, \\alpha)=\\sum_{i=1}^{n} \\alpha_{i}-\\frac{1}{2} \\sum_{i, j=1}^{n} y^{(i)} y^{(j)} \\alpha_{i} \\alpha_{j}\\left(x^{(i)}\\right)^{T} x^{(j)}-b \\sum_{i=1}^{n} \\alpha_{i} y^{(i)} \\\\\n",
    "   \\mathcal{L}(w, b, \\alpha)=\\sum_{i=1}^{n} \\alpha_{i}-\\frac{1}{2} \\sum_{i, j=1}^{n} y^{(i)} y^{(j)} \\alpha_{i} \\alpha_{j}\\left(x^{(i)}\\right)^{T} x^{(j)}\n",
    "   \\end{array}\n",
    "   $$                           \n",
    "   因此：                           \n",
    "   $$\n",
    "   \\begin{aligned}\n",
    "    &\\text { 对拉格朗日化的原问题求最小值, 得到了 } \\mathrm{w} \\text { , 现在可以构造 dual 问題 }\\\\\n",
    "    &\\begin{aligned}\n",
    "    \\max _{\\alpha} & W(\\alpha)=\\sum_{i=1}^{n} \\alpha_{i}-\\frac{1}{2} \\sum_{i, j=1}^{n} y^{(i)} y^{(j)} \\alpha_{i} \\alpha_{j}\\left\\langle x^{(i)}, x^{(j)}\\right\\rangle \\\\\n",
    "    \\text { s.t. } & \\alpha_{i} \\geq 0, \\quad i=1, \\ldots, n \\\\\n",
    "    & \\sum_{i=1}^{n} \\alpha_{i} y^{(i)}=0\n",
    "    \\end{aligned}\\\\\n",
    "    &\\text { 可以推导出 b的值为: } b^{*}=-\\frac{\\max _{i: y^{(i)}=-1} w^{* T} x^{(i)}+\\min _{i: y^{(i)}=1} w^{* T} x^{(i)}}{2}\\\\\n",
    "    &\\begin{array}{r}\n",
    "    \\text { SVM的决策子如下,值的符号为类别. } \\\\\n",
    "    \\qquad w^{T} x+b=\\left(\\sum_{i=1}^{n} \\alpha_{i} y^{(i)} x^{(i)}\\right)^{T} x+b=\\sum_{i=1}^{n} \\alpha_{i} y^{(i)}\\left\\langle x^{(i)}, x\\right\\rangle+b\n",
    "    \\end{array}\n",
    "    \\end{aligned}\n",
    "    $$                              \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "   - 非线性支持向量机：                     \n",
    "   在刚刚的讨论中，我们都是着重讨论了线性支持向量机是如何工作的，但是在现实生活中，我们很难碰到线性可分的数据集，如：             \n",
    "   ![jupyter](./1.29.png)                         \n",
    "   那我们应该如何处理非线性问题呢？答案就是将数据投影至更加高的维度！                    \n",
    "   ![jupyter](./1.30.png)                     \n",
    "   上图中，在一维数据做不到线性可分，我们将数据投影至二维平面就可以成功线性可分。那么，我们来详细探讨下这其中的奥妙：                        \n",
    "   \n",
    "   $$\n",
    "   \\begin{array}{l}\n",
    "   \\Phi: \\mathcal{X} \\mapsto \\hat{\\mathcal{X}}=\\Phi(\\mathbf{x}) \\\\\n",
    "   \\Phi\\left(\\left[x_{i 1}, x_{i 2}\\right]\\right)=\\left[x_{i 1}, x_{i 2}, x_{i 1} x_{i 2}, x_{i 1}^{2}, x_{i 2}^{2}\\right]\n",
    "   \\end{array}\n",
    "   $$                      \n",
    "   如果我们使用上面公式的形式将低维数据拓展至高维数据，则必须面临一个很大的问题，那就是：维度爆炸导致的计算量太大的问题。假如是一个2维特征的数据，我们可以将其映射到5维来做特征的内积，如果原始空间是三维，可以映射到到19维空间，似乎还可以处理。但是如果我们的低维特征是100个维度，1000个维度呢？那么我们要将其映射到超级高的维度来计算特征的内积。这时候映射成的高维维度是爆炸性增长的，这个计算量实在是太大了，而且如果遇到无穷维的情况，就根本无从计算了。能不能呢个避免这个问题呢？核函数隆重登场：                       \n",
    "   回顾线性可分SVM的优化目标函数：                  \n",
    "   $$\n",
    "   \\underbrace{ min }_{\\alpha}  \\frac{1}{2}\\sum\\limits_{i=1,j=1}^{m}\\alpha_i\\alpha_jy_iy_jx_i \\bullet x_j - \\sum\\limits_{i=1}^{m}\\alpha_i\\\\\n",
    "   s.t. \\; \\sum\\limits_{i=1}^{m}\\alpha_iy_i = 0\\\\\n",
    "   0 \\leq \\alpha_i \\leq C\n",
    "   $$                  \n",
    "   注意到上式低维特征仅仅以内积$x_i \\bullet x_j$ 的形式出现，如果我们定义一个低维特征空间到高维特征空间的映射$\\phi$，将所有特征映射到一个更高的维度，让数据线性可分，我们就可以继续按前两篇的方法来优化目标函数，求出分离超平面和分类决策函数了。也就是说现在的SVM的优化目标函数变成：                         \n",
    "   $$\n",
    "   \\begin{array}{c}\n",
    "   \\underbrace{\\min }_{\\alpha} \\frac{1}{2} \\sum_{i=1, j=1}^{m} \\alpha_{i} \\alpha_{j} y_{i} y_{j} \\phi\\left(x_{i}\\right) \\bullet \\phi\\left(x_{j}\\right)-\\sum_{i=1}^{m} \\alpha_{i} \\\\\n",
    "   \\text { s. } t . \\sum_{i=1}^{m} \\alpha_{i} y_{i}=0 \\\\\n",
    "   0 \\leq \\alpha_{i} \\leq C\n",
    "   \\end{array}\n",
    "   $$                    \n",
    "   可以看到，和线性可分SVM的优化目标函数的区别仅仅是将内积$x_i \\bullet x_j$替换为$\\phi(x_i) \\bullet \\phi(x_j)$。我们要将其映射到超级高的维度来计算特征的内积。这时候映射成的高维维度是爆炸性增长的，这个计算量实在是太大了，而且如果遇到无穷维的情况，就根本无从计算了。下面引入核函数：               \n",
    "   假设$\\phi$是一个从低维的输入空间$\\chi$（欧式空间的子集或者离散集合）到高维的希尔伯特空间的$\\mathcal{H}$映射。那么如果存在函数$K(x,z)$，对于任意$x, z \\in \\chi$，都有：                    \n",
    "   $$\n",
    "   K(x, z) = \\phi(x) \\bullet \\phi(z)\n",
    "   $$                       \n",
    "   那么我们就称$K(x, z)$为核函数。                   \n",
    "   仔细发现，$K(x, z)$的计算是在低维特征空间来计算的，它避免了在刚才我们提到了在高维维度空间计算内积的恐怖计算量。也就是说，我们可以好好享受在高维特征空间线性可分的利益，却避免了高维特征空间恐怖的内积计算量。下面介绍几种常用的核函数：                     \n",
    "   (1)  多项式核函数：                   \n",
    "   多项式核函数（Polynomial Kernel）是线性不可分SVM常用的核函数之一，表达式为：                         \n",
    "   $$\n",
    "   K\\left(\\mathbf{x}_{i}, \\mathbf{x}_{j}\\right)=\\left(\\left\\langle\\mathbf{x}_{i}, \\mathbf{x}_{j}\\right\\rangle+c\\right)^{d}\n",
    "   $$             \n",
    "   C用来控制低阶项的强度，C=0,d=1代表无核函数。                       \n",
    "   (2) 高斯核函数：                                                   \n",
    "   高斯核函数（Gaussian Kernel），在SVM中也称为径向基核函数（Radial Basis Function,RBF），它是非线性分类SVM最主流的核函数。libsvm默认的核函数就是它。表达式为：                             \n",
    "   $$\n",
    "   K\\left(\\mathbf{x}_{i}, \\mathbf{x}_{j}\\right)=\\exp \\left(-\\frac{\\left\\|\\mathbf{x}_{i}-\\mathbf{x}_{j}\\right\\|_{2}^{2}}{2 \\sigma^{2}}\\right)\n",
    "   $$                              \n",
    "   使用高斯核函数之前需要将特征标准化，因此这里衡量的是样本之间的相似度。                    \n",
    "   (3) Sigmoid核函数：                    \n",
    "   Sigmoid核函数（Sigmoid Kernel）也是线性不可分SVM常用的核函数之一，表达式为：               \n",
    "   $$\n",
    "   K\\left(\\mathbf{x}_{i}, \\mathbf{x}_{j}\\right)=\\tanh \\left(\\alpha \\mathbf{x}_{i}^{\\top} \\mathbf{x}_{j}+c\\right)\n",
    "   $$                        \n",
    "   此时的SVM相当于没有隐藏层的简单神经网络。                     \n",
    "   (4) 余弦相似度核：                  \n",
    "   常用于衡量两段文字的余弦相似度，表达式为：                    \n",
    "   $$\n",
    "   K\\left(\\mathbf{x}_{i}, \\mathbf{x}_{j}\\right)=\\frac{\\mathbf{x}_{i}^{\\top} \\mathbf{x}_{j}}{\\left\\|\\mathbf{x}_{i}\\right\\|\\left\\|\\mathbf{x}_{j}\\right\\|}\n",
    "   $$\n",
    "   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-14T12:40:54.152402Z",
     "start_time": "2021-03-14T12:40:54.139402Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9733333333333334"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.svm import SVC\n",
    "'''\n",
    "C:正则化参数。正则化的强度与C成反比。必须严格为正。惩罚是平方的l2惩罚。\n",
    "kernel:{'linear'，'poly'，'rbf'，'sigmoid'，'precomputed'}，默认='rbf'\n",
    "degree:多项式和的阶数\n",
    "gamma:“ rbf”，“ poly”和“ Sigmoid”的内核系数。\n",
    "shrinking:是否软间隔分类，默认true\n",
    "\n",
    "'''\n",
    "svc_iris = make_pipeline(StandardScaler(), SVC(gamma='auto'))\n",
    "svc_iris.fit(X, y)\n",
    "svc_iris.score(X,y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(4) 评估模型的性能并调参:                        \n",
    "更详细的可以查看笔者的知乎：https://zhuanlan.zhihu.com/p/140040705"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-14T12:41:04.641260Z",
     "start_time": "2021-03-14T12:41:01.014259Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "网格搜索经历时间：3.611 S\n",
      "0.9800000000000001\n",
      "{'svc__C': 1.0, 'svc__gamma': 0.1, 'svc__kernel': 'rbf'}\n"
     ]
    }
   ],
   "source": [
    "# 使用网格搜索进行超参数调优：\n",
    "# 方式1：网格搜索GridSearchCV()\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.svm import SVC\n",
    "import time\n",
    "\n",
    "start_time = time.time()\n",
    "pipe_svc = make_pipeline(StandardScaler(),SVC(random_state=1))\n",
    "param_range = [0.0001,0.001,0.01,0.1,1.0,10.0,100.0,1000.0]\n",
    "param_grid = [{'svc__C':param_range,'svc__kernel':['linear']},{'svc__C':param_range,'svc__gamma':param_range,'svc__kernel':['rbf']}]\n",
    "gs = GridSearchCV(estimator=pipe_svc,param_grid=param_grid,scoring='accuracy',cv=10,n_jobs=-1)\n",
    "gs = gs.fit(X,y)\n",
    "end_time = time.time()\n",
    "print(\"网格搜索经历时间：%.3f S\" % float(end_time-start_time))\n",
    "print(gs.best_score_)\n",
    "print(gs.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-14T12:41:06.386333Z",
     "start_time": "2021-03-14T12:41:06.243333Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "随机网格搜索经历时间：0.132 S\n",
      "0.9800000000000001\n",
      "{'svc__kernel': 'rbf', 'svc__gamma': 0.1, 'svc__C': 1.0}\n"
     ]
    }
   ],
   "source": [
    "# 方式2：随机网格搜索RandomizedSearchCV()\n",
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "from sklearn.svm import SVC\n",
    "import time\n",
    "\n",
    "start_time = time.time()\n",
    "pipe_svc = make_pipeline(StandardScaler(),SVC(random_state=1))\n",
    "param_range = [0.0001,0.001,0.01,0.1,1.0,10.0,100.0,1000.0]\n",
    "param_grid = [{'svc__C':param_range,'svc__kernel':['linear']},{'svc__C':param_range,'svc__gamma':param_range,'svc__kernel':['rbf']}]\n",
    "# param_grid = [{'svc__C':param_range,'svc__kernel':['linear','rbf'],'svc__gamma':param_range}]\n",
    "gs = RandomizedSearchCV(estimator=pipe_svc, param_distributions=param_grid,scoring='accuracy',cv=10,n_jobs=-1)\n",
    "gs = gs.fit(X,y)\n",
    "end_time = time.time()\n",
    "print(\"随机网格搜索经历时间：%.3f S\" % float(end_time-start_time))\n",
    "print(gs.best_score_)\n",
    "print(gs.best_params_)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**当类别为两类时，可以绘制混淆矩阵与ROC曲线**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-14T12:41:12.903180Z",
     "start_time": "2021-03-14T12:41:10.122598Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAALQAAAC4CAYAAABKD8ZJAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAARwklEQVR4nO3dfVRTZ54H8C8YEFkgQCIovhRrdTQqrYAs4BuRuCpia6kCM3ZcWxlahVLP8bQq7rZltK06Wt+hLkbqy3hEj7Z1RtdWXCoFfMFyXFSqNj3HASySkBcIVMDkZv9wTY3yckGSm3n4fc7xHG9yzf3m+uXyJLm5j4vFYrGAEEa4Ch2AkN5EhSZMoUITplChCVOo0IQpVGjCFJHQAZzV1atXkZeXB47jEBsbi/nz5wsdSTDZ2dkoLy+HWCzGli1bhI7TKTpCt4PjOCiVSmRmZmLr1q0oKSlBTU2N0LEEExMTg8zMTKFj8EKFbodKpcKgQYMQGBgIkUiE6OholJWVCR1LMDKZDF5eXkLH4IUK3Q6dTgeJRGJdlkgk0Ol0AiYifFGhCVOo0O3w9/eHVqu1Lmu1Wvj7+wuYiPBFhW7HyJEjUVtbC7VaDZPJhNLSUoSHhwsdi/DgQmfbta+8vBz79+8Hx3GQy+VISEgQOpJgtm3bhsrKShiNRojFYiQmJmLGjBlCx2oXFZowhYYchClUaMIUKjRhChWaMIUKTZhChe5CQUGB0BGcirPvDyp0F5z9P9DRnH1/UKEJU+iDFcIUp/7GyqGvzgsdAXOmh+K/z5cLHQOzpowXOgIAwFcshqGhQdAMA6WSDu+jIQdhChWaMIUKTZhChSZMoUITplChCVOo0IQpVGjCFCo0YQoVmjCFCk2YQoUmTKFCE6ZQoQlTqNCEKVRowhQqNGEKFZowhQpNmEKFJkyhQhOmUKEJU6jQhClUaMIUKjRhChWaMIUKTZji1Ne2c6Tau1XYtTnLuqyuq8Vrv38Dbfp/4IMPP8QvNf/AR5ty8PwLYwRMKYx3MzJw9uy3kEqlqKi4JnScTjms0FevXkVeXh44jkNsbCzmz5/vqE3zMnjIcHy8VQkA4MxmZKQsQPi/TsXYsSPx7qo/Y1/OFoETCic5ORlLly5Fenqa0FG65JAhB8dxUCqVyMzMxNatW1FSUoKamhpHbLpHblwrR8CgIZAGDMLo3/0Og4cMFzqSoKKio+Hr5yd0DF4cUmiVSoVBgwYhMDAQIpEI0dHRKCsrc8Sme+Ti9/+DqKnOOVMq6ZxDCq3T6SCR/HZNX4lEAp1O54hNd5vpwQOUl5UgIjpG6CikB5zqRWFBQYF1Do8NGzZgzvRQh2c4ffoUwsPCkPyKAgAg9vbEnOmhyPH1RnToGEyc6PhMAODjNUCQ7T5i8PaGq2s/9OvXD75isaBZOuOQQvv7+0Or1VqXtVot/P39n1pPoVBAoVBYl4W4cv7uz/8LIS9FWLf96Ar+OoMRpeU3ca/R4ZEACH8F/0ajERxnhtlspiv4jxw5ErW1tVCr1TCZTCgtLUV4eLgjNt0tLS33cePqDwiPnGq97dTf/4aMlAVQ3arElvVrsCnrPQETCuOt1D8hbs5sqFQqjBgRjL8eOiR0pA45bNKg8vJy7N+/HxzHQS6XIyEhoct/Q3Os/EboI/Qjzj7HisPG0KGhoQgNFWb8SfoO+uibMIUKTZjS4ZDjgw8+gIuLS5cPkJWV1eU6hDhKh4WeMYM+KSP/fDosdExMjANjENI7eL3LYbFYcO7cOZSUlMBoNGLz5s2orKyEwWBAdHS0vTMSwhuvF4X5+fkoLCyEQqFAfX09gIfnY3z99dd2DUdId/Eq9Pnz57Fq1SpMnjzZ+kIxICAAarXaruEI6S5eheY4Dh4eHja3tbS0PHUbIULjVeiJEyfiwIEDePDgAYCHY+r8/HyEhYXZNRwh3cWr0IsXL4Zer8eSJUvw66+/YvHixdBoNFi0aJG98xHSLbze5fD09MR7772HhoYGaDQaSKVS+Pr62jkaId3H++Sk5uZmVFRUQK/Xw8/PDxMnToSXl5c9sxHSbbwKff36dWzevBlBQUGQSqXQarVQKpVYuXIlJkyYYO+MhPDGq9BKpRKpqak2H6JcuHABSqUS27Zts1c2QrqN14tCvV6PyMhIm9siIiJgMBjskYmQHuNV6GnTpuHMmTM2t3377beYNm2aXUIR0lO8Th/lOA5nz57FyZMn4e/vD51Oh4aGBowaNcphQQnhg/fpo7GxsXYPQ8izotNHCVN4vw9tMBigUqlgNBrx+BfF6YsAxJnwKvTly5exc+dODB48GNXV1Rg2bBiqq6sxZswYKjRxKrwKnZ+fj+XLlyMqKgpvvPEGNm3ahMLCQlRXV9s7HyHdwuttu/r6ekRFRdncNn36dBQVFdklFCE9xavQPj4+1g9RBg4ciNu3b6Ourg4cx9kzGyHdxmvIERsbi5s3byIyMhJz585FVlYWXFxcEB8fb+98hHQLr0I/Pn3E9OnTMW7cOLS0tGDo0KH2ykVIj/To2nZSqbS3cxDSKzos9LJly3g9QE5OTq+FIeRZdXg53crKSl4PIJPJejXQ49Qabdcr2Zmfrxh6g7CXjwWAil8EutL6EyJGBeHyT78ImkHx4ogO7+vwCG3PohJiL3T1UcIUKjRhChWaMKVbheY4Dnq93l5ZCHlmvN6Hbm5uxt69e3Hx4kWIRCIcPHgQV65cgUqlQnJysr0zEsIbryN0bm4uPD09kZ2dDZHo4c/A6NGjUVpaatdwhHQXryP0tWvXsGfPHmuZgYcnLDUIPL0XIU/idYT29PSE0Wi0ua2+vh5+fn52CUVIT/EqdGxsLLZs2YLr16/DYrHg9u3b2L17N2bOnGnvfIR0C68hxyuvvAJ3d3colUqYzWbk5ORAoVAgLi7O3vkI6RZehXZxcUFcXBwVmDg93hdr7Mj48c4xBzUhAM9CP3mKaGNjI0wmEyQSCXbt2mWXYIT0BK9C796922aZ4zgcP34cAwYMsEsoQnqqR+dyuLq6IiEhgaZ1I06nxycnVVRUwNWVzm0izoXXkOPJr2O1tbWhra0NKSkpdglFSE/xKvQ777xjs9y/f38MHjwYnp6edglFSE91WWiO43D06FGsXbsWbm5ujshESI91OQh2dXWFWq1GB9+lJcSp8HpVt2DBAuTm5kKj0YDjOJs/hDgTXmPoPXv2AEC7F2fMz8/v3USEPANehaZPA8k/C15DjgsXLmDgwIFP/bl06ZK98xHSLbwKffz48W7dTohQOh1yPDrLjuO4p864q6uro3M5iNPptNCPzrJra2uzOePOxcUFvr6+ePPNN+2bjpBu6vBijY/btWsX0tPTHZHHhlAXa7x79y7S05ejXqNBv34i/GHRIqSmviVIlkeEulij2WxG2h9egTQgEOt3KhHg1oqkpGQ0Nhgwaux4rPp4C9zc3B2aqbOLNfIaQz9rmbOzs5GSkoKVK1c+0+M4ikjUD1lZf8b3xaUoLi5G3j4lbt26JXQsQXx5OA/DR4y0Ln/4n/+BhNffxP6/FcLLxwdnvjwqYLqnOeR0uZiYGGRmZjpiU70iMHAQQkJeBAB4e3tj1OjRuFdbK3Aqx9PU1eLS94WYk5AEALBYLCg6/x2mKeYAAP5t3msoKTwrZMSnOKTQMpkMXl5ejthUr7tz5w6uX7uG0LAwoaM4XM5f1uFPK1bD1eVhTRoNeojFYvT7/+uzSAMHQauuEzLiU+iE5k40NzUhKSkR69Z9DG9vb6HjONTFonPw9ZNgtGyC0FG6pUdzrNhLQUEBCgoKAAAbNmyAn69YsCwPHjzA64t+j0WLFuH11xcJluORiH9x7G+40wdv4YeS77D05Ri0tLTCaGxEfs5f0NjQgNARARCJRLisq8LzwcMRMSrIodk641SFVigUUCgU1mWhpoKwWCx4Jz0NwSOeR0bGu31ySoq4Py5H3B+XAwD+t+wijh3IRWrmp2h78ABbcpSQz56Hbdl7MC5iqsOnqHjmdzn6msuXLuHYsaMo/v57hIeHYYY8BgUFzvXiRyhZ69bj+EEl/n2eHI0GA2a/mih0JBu83od+Vtu2bUNlZSWMRiPEYjESExN5TXpPkwb9hiYN+k2PJg3qTStWrHDEZgihIQdhCxWaMIUKTZhChSZMoUITplChCVOo0IQpVGjCFCo0YQoVmjCFCk2YQoUmTKFCE6ZQoQlTqNCEKVRowhQqNGEKFZowhQpNmEKFJkyhQhOmUKEJU6jQhClUaMIUKjRhChWaMIUKTZhChSZMoUITpjjkcrqEOAodobuwevVqoSM4FWffH1RowhQqNGEKFboLj09ixFdaWhoqKioAACdOnMDnn3/e27GecuPGDbz99tsd3p+YmIh79+51+ThqtRqJiYkwm83t3t/Z/ujq3zqCU82C5Yx6UujHJSQk8Fpv9+7dkEgkSE5Ofqbt2duz7g97oyN0F4Q82pDu65NH6LS0NCgUChQVFcFgMGDSpElISUmBu7s7bty4gZ07d2L27Nk4deoUQkJCkJaWhpMnT+LcuXNobm7G+PHjkZqaap3uuaioCEeOHEFLSwvi4+NttnX06FHcu3cPGRkZAICbN2/i0KFDqKmpwYABA5CUlASTyYTi4mIAwKlTpzBu3DisXr0aOp0O+/btw48//ggPDw/MnTsXcXFxAIC2tjbk5ubiypUr8PX1hVwu5/38y8vLceTIEdTV1cHT0xNyuRyJibbTsxUWFuLYsWOwWCyIj4/Hyy+/DADgOK7TfSG0PlloACguLsbatWvh4eGBjRs34sSJE9Zf9waDAU1NTcjOzobFYsGZM2dQVlaGjz76CD4+PsjLy8PevXuxYsUK1NTUIDc3F2vWrMGoUaNw+PBhaLXtT0en0WjwySefIDU1FZGRkbh//z60Wi2Cg4Nx69YtmyEHx3HYuHEjJk2ahBUrVkCr1WLdunUICgrCSy+9hGPHjqGurg47d+5ES0sLPv30U97PvX///khPT8fQoUNRXV2N9evXIzg4GBEREdZ1rl+/ju3bt0OtViMrKwvBwcEICQnpdF84gz475Jg1axakUim8vLzw6quvoqSkxHqfi4sLEhMT4ebmBnd3d5w9exbJycmQSCRwc3PDwoULcenSJZjNZly8eBFhYWGQyWRwc3NDUlISXFxc2t1mcXExJkyYgClTpkAkEsHb2xvBwcHtrvvzzz+jsbERCxYsgEgkQmBgIGJjY1FaWgoAuHDhAhISEuDl5QWpVIo5c+bwfu7jxo3D8OHD4erqiueeew6TJ09GZWWlzToLFy6Eh4cHhg8fDrlcbt0/ne0LZ9Bnj9BSqdT694EDB0Kn01mXfXx84O7ubl3WaDTYvHmzTVFdXV3R0NAAnU4HiURivd3Dw6PDie61Wi0CAwN55dNoNNDr9ViyZIn1No7jMHbsWACAXq+32e7jz6crP/30Ew4fPoyqqiqYTCaYTCZERkbarPPkY1dVVVlzdbQvnEGfLXR9fb3N3/39/a3LTx5hJRIJli1bhjFjxjz1OH5+frh79651ubW1FUajsd1tSiQSqFSqdu97cptSqRQBAQHYsWNHu+v7+vpCq9Vi2LBhTz2fruzYsQOzZs3CmjVr4O7uji+++AKNjbYz1Wq1WgwZMsT62H5+ftbn0NG+UKvVvDPYS58dcnzzzTfQarVoamrCiRMnEBUV1eG6M2fOxJEjR6DRaAAAjY2NKCsrAwBERkbihx9+wM2bN2EymZCfn4+OTo+ZOnUqrl27htLSUpjNZhiNRty5cwcAIBaLUVdXZ133hRdewIABA/DVV1+hra0NHMehqqrK+gMRFRWFL7/8Ek1NTdBqtThz5gzv537//n14eXnB3d0dKpXK+oL0ccePH0drayuqq6vx3XffITo6ust94Qz67BF6ypQpWL9+PfR6PcLDw/Haa691uO6jdxYerS8WixEVFYVJkyZh2LBhWLp0KbZv347W1lbEx8fb/Lp+nFQqxZo1a3Dw4EHs2bMHnp6eSEpKQnBwMGbMmIHPPvsMS5YsgUwmw/vvv49Vq1bhwIEDSEtLg8lkQlBQEJKSkgA8HOPm5uYiPT0dfn5+kMvlOH36NK/nnpKSggMHDmDfvn2QyWSIiopCc3OzzToymQwZGRngOA7z5s3Diy++2OW+cAZ98my7tLQ0vPXWWwgJCRE6CullfXbIQdhEhSZM6ZNDDsIuOkITplChCVOo0IQpVGjCFCo0YQoVmjDl/wCeuO9su9rQ7gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 180x180 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 混淆矩阵：\n",
    "# 加载数据\n",
    "df = pd.read_csv(\"http://archive.ics.uci.edu/ml/machine-learning-databases/breast-cancer-wisconsin/wdbc.data\",header=None)\n",
    "'''\n",
    "乳腺癌数据集：569个恶性和良性肿瘤细胞的样本，M为恶性，B为良性\n",
    "'''\n",
    "# 做基本的数据预处理\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "\n",
    "X = df.iloc[:,2:].values\n",
    "y = df.iloc[:,1].values\n",
    "le = LabelEncoder()    #将M-B等字符串编码成计算机能识别的0-1\n",
    "y = le.fit_transform(y)\n",
    "le.transform(['M','B'])\n",
    "# 数据切分8：2\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train,X_test,y_train,y_test = train_test_split(X,y,test_size=0.2,stratify=y,random_state=1)\n",
    "from sklearn.svm import SVC\n",
    "pipe_svc = make_pipeline(StandardScaler(),SVC(random_state=1))\n",
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "pipe_svc.fit(X_train,y_train)\n",
    "y_pred = pipe_svc.predict(X_test)\n",
    "confmat = confusion_matrix(y_true=y_test,y_pred=y_pred)\n",
    "fig,ax = plt.subplots(figsize=(2.5,2.5))\n",
    "ax.matshow(confmat, cmap=plt.cm.Blues,alpha=0.3)\n",
    "for i in range(confmat.shape[0]):\n",
    "    for j in range(confmat.shape[1]):\n",
    "        ax.text(x=j,y=i,s=confmat[i,j],va='center',ha='center')\n",
    "plt.xlabel('predicted label')\n",
    "plt.ylabel('true label')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-14T12:41:31.021663Z",
     "start_time": "2021-03-14T12:41:20.711664Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Figure size 432x288 with 0 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAcQAAAFRCAYAAADq9N3vAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABTMElEQVR4nO3dd0BV9f/H8eeFy5Y9RMSJ4iwXbgUHWmlamqPScn0dobl3mmSuzB2W5UpbkvVVKzMTF25xlTMVRw5EAVGQebmf3x/8vN9IxavCvYz34y/uuYdz3vdzL/fF53M+5xyNUkohhBBCFHMW5i5ACCGEKAgkEIUQQggkEIUQQghAAlEIIYQAJBCFEEIIQAJRCCGEACQQhRm0aNGC//znP+Yuo1AqX74806ZNM3cZXLp0CY1Gw+7du81disns2LEDjUbD1atXn2k7BeU9FA/SmrsAYRq9e/dm1apVAFhYWFCqVClatWrFzJkzKV26tElr+e9//4tWKx+93PznP//h/Pnz7NixI8fyqKgo7O3tzVNUIRMcHIyvry9ffvllnmyvSZMmxMTE4OXlZdT68h4WPtJDLEaaN29OTEwMf//9N99++y1Hjx6la9euJq/Dzc0NJyenfN2HUorMzMx83cfTeNaaPD09cXBwyKNqCqaC+r5ZW1vj7e2NhcWzfW0Wh/ewsJJALEbu/0GXLl2awMBABgwYwL59+7h7965hnS1bttC0aVPs7OwoXbo0ffr0IT4+Psd2wsPDqVevHra2tri7u/PSSy9x+/Ztw/OffPIJVatWxdbWlsqVKzN9+nR0Op3h+X8OmS5duhRnZ2fS0tJy7OOjjz6ibNmy6PV6AM6fP89rr72Gi4sLrq6utG3bluPHjxvW//LLL9FqtWzfvp06depgY2NDRETEQ9shKSmJgQMH4unpiY2NDQEBAfz++++G5+8PB3799de0bt0aOzs7KlasyJo1a3JsJzY2lt69e+Pp6YmjoyNNmzYlMjLS8Pz9IbaNGzfSrFkzbG1tWbZsGbdv36Znz56ULVsWOzs7qlSpwty5c7l/0ajQ0FCWL1/Ozp070Wg0aDQaQy/n38Nt5cuX5/3332fYsGG4ublRsmRJRowYkaO9U1NTGTBgAM7Ozri6uhISEsKECROoVKnSQ9vnvuTkZIYPH06ZMmWwsbGhfPnyzJgxI8c6169f5+WXX8be3p6KFSs+0BtbuHAhtWvXpkSJEnh7e/P6668TExPzzG1036M+i71792br1q2sWrXK0Ib3e2pP+779e8g0MzOTkSNH4uvri42NDaVKleL1119/4vdQp9PxwQcf4Ofnh42NDaVLl+bdd9/N9b0R+USJYqFXr16qdevWhsfXrl1TgYGBytLSUiUnJyullNq6dauys7NTixYtUmfPnlUHDx5ULVq0UIGBgUqv1yullFqxYoXSarVq6tSp6uTJk+qPP/5QCxYsULdu3VJKKTVlyhRVtmxZ9d///ldduHBBbdy4UZUpU0ZNmjTJsO+goCDVr18/pZRSiYmJytbWVq1ZsyZHvdWrV1cTJkxQSil148YNVbJkSTVo0CD1559/qjNnzqghQ4YoNzc3dfPmTaWUUitXrlQajUbVr19fbdu2TUVHRxue+7cuXbqocuXKqd9++02dOnVKDR06VFlZWanTp08rpZS6ePGiAlSpUqXU119/rc6cOaPee+89ZWFhoY4cOaKUUiolJUVVq1ZNde7cWUVFRalz586padOmKWtra3Xq1CmllFLbt29XgKpSpYr66aef1IULF9SVK1dUTEyMmjlzpjp8+LC6cOGC+uqrr5SDg4NasWKFUkqppKQk9eabb6rGjRurmJgYFRMTo1JSUpRSSpUrV059+OGHhtdSrlw55eLiombOnKnOnj2rwsPDlVarVcuWLTOs8+677yovLy+1YcMGdebMGTV+/Hjl5OSk/Pz8Hvl50ev1KigoSFWoUEGtW7dORUdHq507d6ovvvgiRxtVqFBBhYeHq3PnzqkJEyYoS0tL9ddffxm2s2DBArVlyxZ14cIFtXfvXtW4cWMVGBhoeP5p2+hxn8XExETVvHlz1a1bN0MbpqenP9P7dn/5lStXlFJKzZ07V5UuXVpt375dXb58WR08eFDNnz//id/Dt99+W3l6eqrVq1er8+fPq3379ql58+Y98r0R+UcCsZjo1auXsrS0VA4ODsrOzk4BClCjRo0yrBMUFKTGjRuX4/cuX76sAHX06FGllFJlypRRgwcPfug+7t27p+zs7NSmTZtyLF+1apVydnbOsZ/7gaiUUt27d1ft2rUzPI6KilKAOnPmjFIqO2QbNmyYY5t6vV5VrFjR8AW0cuVKBajIyMhc2+HcuXMKUBs3bsyxvE6dOqpPnz5Kqf992f8zxJVSqnHjxqpnz56G/ZUuXVplZmbmWKdly5Zq2LBhSqn/fbGuXr0615qUUmro0KEqODjY8Lhfv34qKCjogfUeFogdOnTIsc6LL76oXn/9daWUUsnJycra2jpHQCqlVMOGDXMNxIiICAWoqKiohz5/v43mzp1rWKbT6VSJEiXUkiVLHrndI0eOKEBdvXpVKfVsbZTbZ1EppVq3bq169eqVY9mzvG//DsShQ4eqli1bGv5Z/Ddj3sP7n8e1a9c+8nUI05GZDcVIw4YNWbVqFWlpaXz//fdERETkGLqJiopi//79hIWFPfC7586dw8fHhytXrtC2bduHbv/kyZOkpqby2muvodFoDMuzsrJIS0vj1q1beHp6PvB7vXr1omPHjty8eRMvLy9Wr15NgwYNqFKliqGuw4cPU6JEiRy/l5qayrlz53Isq1+/fq5tcOrUKQACAwNzLA8MDGTfvn05ljVu3DjH46ZNm7J161ZDTTdu3MDFxSXHOunp6djZ2eVY1qBBgxyP9Xo9s2fPZs2aNVy9epW0tDQyMzMpV65crrU/Su3atXM89vHx4eLFi0D2UHNGRgaNGjV64LX9/PPPj9zm4cOHcXV1JSAgwOh9W1pa4uXlRWxsrGHZjh07mDlzJqdOnSIxMdEwBH758uUck7metI1u3ryZ62fxUZ7lffu3Pn360KZNGypVqkSbNm1o06YNHTp0wNra2uh6jhw5AvDEr0PkDwnEYsTOzs5w3KhmzZpER0fz7rvvsnTpUiD7S2jcuHG89dZbD/yut7c3KSkpuW7//pfd2rVr8ff3f+B5Nze3h/5e27Zt8fDw4Ntvv2Xw4MGsWbOG0NDQHNtt3br1Q4Pa2dnZ8LOlpSW2tra51phX9Ho91apVY926dQ889+8ZhP+eQDF37lxmzpzJ/PnzqVOnDo6OjsyfP5+NGzc+VS3//gLWaDSG9+Kfy/JDbvv++++/adeuHW+99Rbvv/8+Hh4eXL16leDgYDIyMnL8Xn630X3P8r79W+3atbl48SJbtmxh+/btDBs2jMmTJ7N///58nzQm8ocEYjEWGhpKtWrVGDhwIAEBAQQEBHDy5MlHTrYoUaIEvr6+/P7773Ts2PGB52vUqIGtrS0XLlygXbt2RtdhaWlJjx49+Oqrr6hYsSJ37twxTE4ACAgI4Msvv8TX1/eZA69GjRoAREZG5qgxMjKSOnXq5Fh3//79OdbZu3cv1atXN9S0evVqnJycjJ6G/899vfjii/Tt29ew7N89XWtra7Kysp5ouw9TqVIlrK2t2bdvn6F2yH5tualXrx63b9/m0KFDj+0lPkpUVBSpqaksWLDA0Ps6fPiwUb/7uDby8vLK9bMID2/DZ3nfHqZEiRJ06tSJTp06MXHiREqVKsXOnTsNPcXHvYd169YF4Pfff6dLly7PXI94NjLLtBirXLkyHTp04L333gNg6tSpbNiwgZEjR3Ls2DGio6P57bff6NevH6mpqQBMmTKFzz//nA8//JDTp09z8uRJwsLCiIuLo0SJEkycOJGJEyeyePFi/vrrL06ePMmaNWsYN25crrW8/fbbHDlyhClTpvDyyy/n6E0OGTKErKwsXnnlFXbt2sWlS5fYvXs37733Hnv37n2i1+zn50fXrl0JCQlh8+bNnDlzhmHDhnHixAnGjBmTY93ly5fz7bffcvbsWd5//3327dvHyJEjAejRowcVKlSgffv2/P7771y6dIkDBw4wc+ZM1q9fn2sNVapUYceOHWzfvp2zZ88yadIkDhw4kGOdChUqcObMGU6ePElcXBzp6elP9Drvc3BwYODAgUyaNIlffvmFs2fP8t5773H69Olce42tWrWiefPmdO/enQ0bNnDx4kX27NnDsmXLjN535cqV0Wg0zJ07l4sXL7J+/XqmTp1q1O8a00a5fRYhuw0PHz5MdHQ0cXFxZGZmPtP79m8ff/wx33zzDSdPnuTixYusWLECS0tLw+iIMe9hpUqV6NGjByEhIXz99ddER0cTFRXFwoULn6gWkUfMfRBTmMa/Z5net2fPHgWo7du3K6WUioyMVK1bt1YlSpRQ9vb2qmrVqmrYsGE5JiF8/fXX6vnnn1fW1tbKzc1NtWvXTt2+fdvw/NKlS1WtWrWUjY2NcnFxUQ0aNFCffvqp4fl/T6q5r3bt2gpQ69evf+C5S5cuqTfffFN5eHgoa2trVbZsWdWjRw914cIFpVT2ZAlLS0uj2uLOnTtqwIABhm3Vq1dPbd682fD8/Qkjq1evVkFBQcrGxkaVL19effPNNzm2ExcXpwYNGqR8fHyUlZWV8vHxUa+++qphJuq/J2Hcl5iYqLp27aocHR2Vm5ubCgkJUZMmTVLlypUzrBMfH69eeukl5eTkpAC1cuVKpdTDJ9X887FSD07mSElJUf3791eOjo7K2dlZvfPOO2rYsGGqZs2aubbT3bt31ZAhQ5S3t7eysrJS5cuXVzNnzszRRrt27crxO35+fmrKlCmGx2FhYcrX11fZ2tqqpk2bqk2bNuX4vD1LGymV+2cxOjpaNW/eXDk4OOTY59O+b/9evmTJElW3bl3l6OioHBwcVEBAQI7PrrHvYUZGhuG1WVlZqdKlSxsm+AjT0ij1rxN7hCjmLl26RIUKFdi1axfNmjUzdzn5olWrVri6uvLjjz+auxQhCgw5hihEEXf8+HGOHDlC48aNycjI4KuvvmL79u1s2rTJ3KUJUaBIIApRxGk0Gj777DOGDh2KXq+natWqrFu3jhdffNHcpQlRoMiQqRBCCIHMMhVCCCEACUQhhBACkEAUQgghgCIwqeb69et5sh0PDw/DCb3i0aSdjCPtZDxpK+NIOxnPx8fnqX5PeohCCCEEEohCCCEEIIEohBBCABKIQgghBCCBKIQQQgASiEIIIQQggSiEEEIAJjoP8dNPP+XIkSM4Ozszd+7cB55XSrFy5UqOHj2KjY0NISEhVKxY0RSlCSGEEICJeogtWrRg4sSJj3z+6NGj3Lhxg0WLFjFgwIAnuiu3EEIIkRdM0kOsXr06N2/efOTzhw4dIjAwEI1Gg7+/P/fu3eP27du4urqaojyTcdv5FrbXt5m7jGf2dNeAKH6knYwnbWUcaafcXU5wRq80MOP2U/1+gbh0W0JCAh4eHobH7u7uJCQkPDQQIyIiiIiIAGDWrFk5fu9ZaLXaPNvWo1gXgTAUQoiCSCnotOp17K0y2T3j6bZRIALxSQQHBxMcHGx4nFfX9jPFdQLv/3d3/Y1r+bqf/CTXUzSOtJPxpK2MI+30eJP9Yli+/ORT/36BCEQ3N7ccb3R8fDxubm5mrEgIIURBptcrvvrqNJcu3WXKlEYANGxYioYNSz31NgvEaRcBAQFERkailOLs2bPY29sXueOHQggh8sbly3fp1m0jEyfu4YsvjnPyZHyebNckPcQFCxZw6tQpkpKSGDRoEN26dUOn0wHQtm1b6tSpw5EjRxg6dCjW1taEhISYoiwhhBCFiF6vWLXqFNOnHyQ1VYe7uy0zZjSlRg33PNm+SQJx+PDhuT6v0Wj4z3/+Y4pSjFZUZoQKIURRcOnSXUaPjmTfvhgAXnnFjw8/bIy7u12e7aNAHEMsiPIrDNN8WuXLdoUQoigLCzvGvn0xeHjYMXNmU9q1q5Dn+5BAfIzCPCNUCCEKM71eYWGhAeC99xpgZWXBmDEBuLnZ5sv+CsSkGiGEEOI+vV6xdOlxOnTYQHp6FgCurrbMnNks38IQpIcohBCiAImOTmTUqEiiomIB2Lz5Eh07+plk3xKIQgghzC4rS8+yZSeYPfsQaWlZeHnZ8dFHzWnbtpzJapBAJHtGqfX1bXKdQCGEMIPz57N7hYcOZfcKu3SpTGhoI1xd82949GEkEHn0jFKZESqEEPnvyJGbHDoUS8mS9nz0UTPatDFdr/CfJBD/QWaUCiGEaSQnZ1CihDUAXbtWJjExnW7d/HFxsTFbTTLLVAghhMlkZelZsuRPGjZcw/nziUD2xVkGDHjOrGEI0kMUQghhIufO3WbkyEiOHMm+P+6WLZepVMnFvEX9gwSiEEKIfKXT6fnii+PMmXOY9PQsvL0d+Pjj5rRqVcbcpeUggSiEECLfREcnMmzYDo4evQXAG29U4f33G+HkZG3myh4kgSiEECLfKAWnTydQqpQDc+Y0p0WLgtUr/CcJRCGEEHnq8uW7lC3riEajoVIlF1aubEvt2l4Fslf4TzLLVAghRJ7IzNSzcOFRgoLW8v335wzLAwN9C3wYgvQQhRBC5IHTpxMYMWInx4/HAXD27G0zV/TkJBCFEEI8tcxMPWFhx1i48CiZmXpKly7BnDnNCQz0NXdpT0wCUQghxFO5fj2ZPn1+58SJeADeeqsakyY1MFyBprCRQBRCCPFU3N3tyMzU4+tbgjlzAmnevLS5S3omEohCCCGMdvJkPD4+Dri62mJjY8ny5W3w9LQrtL3Cf5JZpkIIIR4rIyOLuXMP067dOqZM2WdYXqGCc5EIQ5AeohBCiMc4cSKOESN2cupUAgBOTtZkZemxtCxafSoJRCGEEA+VkZHFwoVHCQs7hk6nKFfOkblzg2jcuJS5S8sXEohCCCEekJqqo0OHDZw+nd0r7NevBuPH18fe3srMleUfCUQhhBAPsLPTUq+eF6mpOubNC6Rhw6LZK/wnCUQhhBAA/PHHLbKyFHXregHw/vuNsLDQYGdXPKKieLxKIYQQj5SensW8eUf47LM/KFPGkS1bOmNvb4WDQ9EdHn0YCUQhhCjGjh69yciROzl7NhGNBtq2LYdGozF3WWYhgSiEEMVQWpru/3uFf6LXKypWdGbevCDq1y9p7tLMRgJRCCGKGaUUPXv+xr59MWg0MGjQ84weXa/YHCt8lOL96oUQohjSaDT06lWdmzdTmDcviICA4tsr/CcJRCGEKAYOHYrlzJkEevasBkCHDhV54YVyWFtbmrmygkMCUQghirDUVB0ff3yIL744jlZrQUBASapWdQOQMPwXCUQhhCiioqJiGTlyJxcu3MHCQsPAgc9RvryTucsqsCQQhRCiiElN1TF79iGWLj2OUuDv78K8eUHUqeNl7tIKNAlEIYQoYiZP3st33/2FpaWGkJBajBhRFxsbGR59HAlEIYQoYoYNq8Nff91m2rQm1Krlae5yCo2idTMrIYQohg4ciGHkyJ3o9QqAMmUc+emnjhKGT0h6iEIIUUilpGQya1YUK1acRClo0sSHLl0qAxTby689CwlEIYQohPbvj2HUqEguXbqLVqthyJDadOxY0dxlFWomC8Rjx46xcuVK9Ho9rVu35tVXX83xfFxcHIsXL+bevXvo9XrefPNN6tata6ryhBCiULh3L5OZMw+ycuUpAKpVc2PBgiBq1vQwc2WFn0kCUa/Xs3z5ciZNmoS7uzsTJkwgICAAX19fwzo//vgjjRs3pm3btly9epWZM2dKIAohxL+sWfMXK1eeQqvVMHRoHd59t7acYJ9HTBKI58+fx9vbm5Ils6+X16RJE6KionIEokajISUlBYCUlBRcXV1NUZoQQhR4SinDz716Vef48Tj+85/nqFnT3YxVFT0mmWWakJCAu/v/3jh3d3cSEhJyrNO1a1d27drFoEGDmDlzJn379jVFaUIIUaDt3n2N9u3XExt7DwCt1oIFC1pIGOaDAjOpZs+ePbRo0YIOHTpw9uxZPvnkE+bOnYuFRc7MjoiIICIiAoBZs2bh4ZF34+Z5ua2iSqvVSjsZQdrJeNJWD5eUlM6ECdtZuvQoAAsXRjFjRgvzFlXEmSQQ3dzciI+PNzyOj4/Hzc0txzrbtm1j4sSJAPj7+5OZmUlSUhLOzs451gsODiY4ONjwOC4u7pnr88nDbRV1Hh4e0k5GkHYynrTVgyIjrzFmTCRXryZjZWXBsGF1CA1tLu1kJB8fn8ev9BAmGTL18/MjJiaGmzdvotPp2Lt3LwEBATnW8fDw4MSJEwBcvXqVzMxMnJzkIrRCiOIjKSmDsWN38cYbv3L1ajLPPefBr7++yogRdbGykokz+c0kPURLS0v69u3L9OnT0ev1tGzZkjJlyhAeHo6fnx8BAQG8/fbbfP7552zcuBGAkJAQObFUCFGs/PXXbb799gxWVhaMGFGXkJBaWFnJBcVMRaP+OX2pELp+/fozb8Pnu9LZ23rj2jNvq6iT4S3jSDsZr7i3VXp6Vo4Lb69ceZLGjUsZ7ll4X3FvpydRoIdMhRBCPGjHjis0axbOzp1XDcv69KnxQBgK05BAFEIIE7t7N4PRoyPp0eM3rl+/x9dfnzZ3SYICdNqFEEIUB9u2XWHs2F3ExNzDxsaS0aPrMWDAc+YuSyCBKIQQJnH3bgahofsIDz8LQJ06XsyfH0jlynJVroJCAlEIIUxAr1fs3HkVGxtLxo4NoH//mlhaylGrgkQCUQgh8kliYjq2tpbY2mpxcbFh8eJWeHjYUamSi7lLEw8h/54IIUQ+2LLlMq1a/cC8eUcMyxo1KiVhWIBJD1EIIfLQ7dtpTJmyjx9/PA/AoUOx6HR6tFrpfxR0EohCCJFHfv/9MuPG7eLmzVRsbS0ZN64+/frVkGOFhYQEohBCPKO0NB1jxuziv//N7hXWr1+SefOCqFjR+TG/KQoSCUQhhHhGNjaW3L2bga2tJRMmNKBPn+rSKyyEJBCFEOIpJCSkkZycQdmyTmg0GmbPbs69e5nSKyzE5F8YIYR4Qr/+epGWLX8gJGQ7WVl6AEqWtJcwLOQkEIUQwkgJCWm8885W+vePIC4ue+LMnTsZ5i5L5BEZMhVCCCNs3HiRCRN2Ex+fhr29lvfea8Dbb1fHwkLu21pUSCAKIcRjjBixk++/z74GaePGpZg7N5By5ZzMXJXIaxKIQgjxGNWru2Fvr2XSpIa89VY16RUWUUYH4p9//smePXu4c+cO48ePJzo6mtTUVGrWrJmf9QkhhMnFxaVy+nQCzZuXBqBv3xq0a1eB0qVLmLkykZ+MmlSzadMmli5dSqlSpTh9OvtGltbW1qxZsyZfixNCCFNSSrFhQzQtW/5A//5buHYtGQBLSwsJw2LAqB7ir7/+yuTJk/Hy8mLDhg0AlC5dmuvXr+drcUIIYSq3bqUwceIefv31EoChdyiKD6MCMTU1FQ8PjxzLdDodWq0cghRCFG73e4WTJu3l9u10SpSw4v33G/Hmm1XQaORYYXFi1JBptWrVWL9+fY5lmzZtokaNGvlRkxBCmMyMGQcZPHg7t2+nExRUmm3butCjR1UJw2LIqEDs27cvBw8eZPDgwaSlpTFs2DD27dtHr1698rs+IYTIVx07+uHiYsOcOc355puX5FhhMWbUmKerqyszZ84kOjqaW7du4e7uTqVKlbCwkAvdCCEKl9jYFH76KZr+/Z8D4LnnPIiKegN7eyszVybMzahEmz17NhqNhkqVKtG4cWP8/f2xsLBgzpw5+V2fEELkCaUUP/xwjpYt1xIaup9Nmy4anpMwFGBkD/HkyZNPtFwIIQqSGzfuMW7cbiIi/gagZUtfatXyNHNVoqDJNRDDw8OB7Bml93++LzY2Fk9P+UAJIQoupRRr154jNHQfd+5k4ORkTWhoI7p185dJM+IBuQZifHw8AHq93vDzfR4eHnTr1i3/KhNCiGe0evVpJk7cA0CrVmX46KNm+PjIpBnxcLkGYkhICAD+/v4EBwebpCAhhMgrXbpU5ttvz9CvX026dq0svUKRK6OOId4Pw9TUVJKSklBKGZ4rWbJk/lQmhBBP6Pr1ZObPP0JoaGMcHKxwcLBi06ZOcjFuYRSjAvHq1assWrSIy5cvP/Dcv48tCiGEqSmlCA8/S2joPpKSMnFysmHy5IYAEobCaEaddrFs2TJq1KjBihUrsLe3Z+XKlbRp04bBgwfnd31CCJGra9eS6dnzN0aNiiQpKZO2bcsxYMBz5i5LFEJGBeLly5fp0aMHDg4OKKWwt7enZ8+e0jsUQpiNUopvvz1Dq1Y/sGPHVVxcbAgLa8mKFW0oWdLe3OWJQsioIVMrKyuysrLQarU4OjoSFxeHg4MDycnJ+V2fEEI81P79NxgzZhcAL75Yjpkzm+HlJUEonp5RgVi1alX27dtHixYtaNSoETNmzMDKykou7i2EMJvGjUvx9tvVaNSoFB07VpQZpOKZGRWII0eONPz8xhtvUKZMGdLS0ggKCsq3woQQ4p+uXk1iwoQ9jBsXQM2a2bejmzmzmZmrEkXJE9/Q0MLCgsDAQHQ6HREREbz44ov5UZcQQgDZxwq/+uo006Yd5N69TDIysggPb2/uskQR9NhAPH78OJcuXcLb25v69euTlZXF5s2b2bBhAyVKlJBAFELkm7//vsvo0bvYs+c6AO3aVWDGjCZmrkoUVbkG4vr16/nxxx8pU6YMV65c4YUXXuDkyZNYWVkxcOBA6tata6o6hRDFiF6vWL36NNOnHyAlRYebmy0zZjSlQ4eK5i5NFGG5BmJERAQffPABFStW5OzZs0yePJm3336b9u1luEIIkX9u3Upl5syDpKTo6NChItOnN8Hd3c7cZYkiLtdATEpKomLF7P/I/P39sbKyol27dk+1o2PHjrFy5Ur0ej2tW7fm1VdffWCdvXv3snbtWjQaDeXKlWPYsGFPtS8hROGj12dfEtLCQkPJkvbMnNkMa2sLXn5ZeoXCNB57DFEpZbh2qZVV9k009Xq94XkLi8ef26/X61m+fDmTJk3C3d2dCRMmEBAQgK+vr2GdmJgY1q9fz4cffkiJEiW4c+fOE78YIUThdOnSXUaN2kn79hXo27cmAJ07VzJzVaK4yTUQ09LSeP3113Ms+/djY65Wc/78eby9vQ0XAm/SpAlRUVE5AnHr1q288MILlCiRfWsWZ2dn416BEKLQ0usVYWFRTJq0g9RUHdev3+Ott6pjZWXURbSEyFO5BmJYWFie7CQhIQF3d3fDY3d3d86dO5djnevXs2eRTZ48Gb1eT9euXaldu3ae7F8IUfBcvHiHUaMiOXDgBgCdOvkxdWoTCUNhNrkGoqenp6nqQK/XExMTw5QpU0hISGDKlCnMmTMHBweHHOtFREQQEREBwKxZs/Dw8MizGvJyW0WVVquVdjKCtNOj6fWKxYsPMXlydq+wZEkHPvnkRV55xd/cpRVo8pnKf098Yv7TcHNzIz4+3vA4Pj4eNze3B9apXLkyWq0WLy8vSpUqRUxMDJUq5TyOEBwcnONmxXFxcc9cn08ebquo8/DwkHYygrTTo+l0er7++g9SU3V07lyJxYtfRq+/J+31GPKZMp6Pj8/jV3oIk4xN+Pn5ERMTw82bN9HpdOzdu5eAgIAc6zRo0ICTJ08CcPfuXWJiYuTmw0IUEVlZepKSMgDQai2YPz+IlSvb8sknLXFzk9MpRMFgkh6ipaUlffv2Zfr06ej1elq2bEmZMmUIDw/Hz8+PgIAAatWqxR9//MGIESOwsLCgZ8+eODo6mqI8IUQ+On8+kVGjInFzs2XFijZoNBr8/V3x93c1d2lC5KBR98+pMEJcXBwJCQn4+xecsf77k3Gehc93pbO39ca1Z95WUSfDNsaRdsruFS5deoKPPz5EWloW3t72bNz4Kt7eOecFSFsZR9rJeE87ZGpUDzEuLo6FCxdy6dIlAL766iv279/PsWPHGDRo0FPtWAhRdJ0/n8iIETs5cuQmAN26+TNlSiNcXGzMXJkQj2bUMcQvvviCOnXqsGrVKrTa7Ax9/vnn+fPPP/O1OCFE4fP553/Stu1/OXLkJt7eDqxe/QLz5wdJGIoCz6hAPH/+PK+++mqOq9LY29uTkpKSb4UJIQqnW7dSSU/P4vXX/dm27TVaty5r7pKEMIpRQ6bOzs7cuHEjx7js1atX5ZwYIQQ6nZ4rV5KoUCH76lKjRtUjKMiX5s1Lm7kyIZ6MUT3EDh068NFHH7F9+3b0ej27d+9m/vz5vPLKK/ldnxCiAPvrrwReeeUnunTZyJ076QDY2WklDEWhZFQPsVWrVjg6OhIREYG7uzuRkZF0796dBg0a5Hd9QogCSKfT8+mnfzB//hEyMvSUKuXAlSvJODvLcUJReBkViHq9nvr161O/fv38rkcIUcCdOZPAiBE7+fPP7FMAevSoyqRJDXFysjZzZUI8G6MCsX///jRu3JhmzZpRtWrV/K5JCFFArVp1iilT9pGZqcfHx4E5cwIJCvJ9/C8KUQgYFYiTJk1iz549LFy4EAsLC5o2bUqzZs0oW1ZmjwlRnJQp40hmpp4ePaoyeXJDHB2lVyiKjie6Ug3AqVOn2L17NwcOHMDV1ZU5c+bkV21GkSvVmJZcLcM4RaWdMjP17NsXQ2Dg/ybJnD17O08vu1ZU2iq/STsZz2QX9/bx8cHX1xcPDw9u3br1VDsVQhR8p07F8/LL6+nRYxOHDsUalss1SEVRZdSQ6b179zhw4AC7d+/m3LlzPP/887zyyisP3LFCCFH4ZWbqCQs7xoIFR9DpFGXKlOAJB5KEKJSMCsSBAwdSpUoVmjVrxqhRox64aa8Qomg4cSKekSN3cvJk9v1Le/euzsSJDXBwsDJzZULkP6MC8ZNPPsHVVYZJhCjKfvopmnff3Y5Opyhb1pG5cwNp0uTpjsUIURg9MhBPnTpF9erVAbh27RrXrj18wknNmjXzpzIhhEk1alQKR0drOnWqxIQJ9bG3l16hKF4eGYjLly9n7ty5AHz22WcPXUej0RAWFpY/lQkh8lVGRhbffnuGnj2rodVa4OVlz65d3XB1tTV3aUKYxSMD8X4YAixevNgkxQghTOPPP28xcmQkp08nkJycyZAhtQEkDEWxZtRpF7Nnz37ocnOfgyiEeDLp6Vl89FEUL7+8gdOnEyhf3omAgJLmLkuIAsGoSTUnT558ouVCiILnjz9uMWLETv766zYaDfznPzUZP74+dnZGfQ0IUeTl+pcQHh4OgE6nM/x8X2xsLJ6envlXmRAizxw+HEunTj+TlaWoUMGJefOCaNDA29xlCVGg5BqI8fHZ5yLp9XrDz/d5eHjQrVu3/KtMCJFn6tTxomFDb2rW9GDs2ADpFQrxELn+VYSEhADg7+9PcHCwSQoSQjy7tDQdCxcepUePqvj6OmJhoeG779qh1T7x1RqFKDYeGYg3b97Ey8sLgOeee47Y2NiHrleypByQF6IgOXLkJiNG7OT8+USOH4/j669fApAwFOIxHhmIo0ePZvXq1QAMHTr0kRv497FFIYR5pKbqmDv3MJ9/fhy9XlGpkgsjRtQ1d1lCFBqPDMT7YQgSekIUdIcOxTJy5E6io+9gYaEhJOR5Ro2qh62tHCsUwlhP9dcSGxuLRqMxDKkKIczn1q0UunXbSHp6FpUruzBvXhB168rfphBPyqhAXLBgAS+99BJVqlRh+/btLFu2DAsLC/r06UOrVq3yu0YhRC48Pe15993apKXpGDGirvQKhXhKRv3lnDhxgiFDhgDwyy+/MHnyZBwcHPj4448lEIUwsdRUHbNnHyIgoCTt21cAkGOFQuQBowJRp9Oh1WpJSEggOTmZqlWrAnDnzp18LU4IkdPBgzcYOXInFy/eZcOGaIKDy2JjY2nusoQoEowKxPLly7Nu3Tpu3bpF3brZ/4kmJCRgZ2eXr8UJIbKlpuqYNSuK5ctPoBRUrerKvHlBEoZC5CGjTkwaNGgQf//9NxkZGXTv3h2As2fP0qxZs3wtTggBBw7EEBz8I8uWncDCQsOwYXX49ddO1Koll04UIi8Z1UP09vZm2LBhOZY1atSIRo0a5UtRQohsOp2ekSMjuXTpLtWquTFvXiDPPy9BKER+MHo62vbt24mMjCQhIQE3NzcCAwNp2bJlftYmRLGllEKj0aDVWjBnTiC7d19j2LA6WFvLEKkQ+cWoQPzvf//Lzp076dChAx4eHsTFxfHTTz9x+/ZtOnfunN81ClFs3LuXyYwZBwGYPr0pAI0bl6Jx41LmLEuIYsGoQNy6dSuhoaE5bvdUq1YtpkyZIoEoRB7Zs+c6o0bt5MqVZKysLAgJqUXp0iXMXZYQxYZRgZieno6Tk1OOZY6OjmRkZORLUUIUJ/fuZTJt2gFWrz4NQPXqbsyf30LCUAgTM2qWae3atVm0aBHXr18nIyODa9euERYWRq1atfK7PiGKtF27rtG69Q+sXn0arVbD6NH1+PXXTtSs6W7u0oQodozqIfbt25cVK1YwevRosrKy0Gq1NG7cmD59+uR3fUIUaeHhf3HlSjI1a7ozb14QNWpIEAphLo8NxJSUFG7cuEG/fv0ICQkhKSkJR0dHLCzk3mpCPI3UVJ3hjvVTpzahenV3+vd/Disr+ZsSwpxyDcQjR44wf/58MjIysLW1ZcyYMdSsWdNUtQlRpCQlZTBt2gEOH77Jr7++irW1JW5utoSEyKEHIQqCXP8lDQ8Pp0ePHqxevZru3buzZs0aU9UlRJESGXmV1q1/5OuvzxAdncjhwzfNXZIQ4l9yDcTY2FhefPFFbGxseOGFF7hx48ZT7+jYsWMMGzaMd999l/Xr1z9yvf3799OtWzeio6Ofel9CFBR372Ywduwu3nhjE9euJVOrlgebNnWS8wqFKIByHTJVShl+trS0JCsr66l2otfrWb58OZMmTcLd3Z0JEyYQEBCAr69vjvVSU1PZtGkTlStXfqr9CFGQREZeZeTISGJi7mFtbcGoUfUYNOh5tFo5VihEQZRrIKanpzNlyhTD47S0tByPAT744IPH7uT8+fN4e3tTsmRJAJo0aUJUVNQDgRgeHs4rr7zCTz/9ZPQLEKKgun79HjEx96hTx5N584Lw93c1d0lCiFzkGoiDBg3K8fhpr12akJCAu/v/ppO7u7tz7ty5HOtcuHCBuLg46tatK4EoCq1r15INJ9R37+6PnZ2W9u0rSK9QiEIg10Bs0aKFSYrQ6/WsXr2akJCQx64bERFBREQEALNmzcLDwyPP6sjLbRVVWq1W2ukhEhPTGDt2K99/f4qoqH54e2vx9PSkXz+5M8XjyGfKONJO+c/ou108Czc3N+Lj4w2P4+PjcXNzMzxOS0vjypUrhuHXxMREZs+ezdixY/Hz88uxreDgYIKDgw2P4+Linrk+nzzcVlF3/+Lu4n8iIv5m3Lhd3LiRgo2NJZGR56hc2U3ayUjymTKOtJPxfHx8Hr/SQ5gkEP38/IiJieHmzZu4ubmxd+9ehg4danje3t6e5cuXGx6Hhoby1ltvPRCGQhQkiYnphIbuY+3a7OH/unW9mD8/iEqVXMxbmBDiqZgkEC0tLenbty/Tp09Hr9fTsmVLypQpQ3h4OH5+fgQEBJiiDCHyzP79MYSEbCM2NgVbW0vGjAmgf/+aWFrKsUIhCiuTBCJA3bp1qVu3bo5l3bt3f+i6oaGhJqhIiKfn4WFHYmI6AQElmTs3UHqFQhQBRgViZmYmP/zwA3v27CEpKYlVq1bxxx9/EBMTw4svvpjfNQpRIERFxRIQ4IVGo6FSJRfWretAzZru0isUoogw6i951apVXLlyhaFDh6LRaAAoU6YMv//+e74WJ0RBkJCQxrvvbufVV3/ixx/PG5bXquUpYShEEWJUD/HgwYMsWrQIW1tbQyC6ubmRkJCQr8UJYW6//XaJ8eN3c+tWKra2lqSnP93VmoQQBZ9RgajVatHr9TmW3b17F0dHx3wpSghzS0hIY/Lkvaxfn31N3UaNvJkzJ5AKFZzNXJkQIr8YNd7TqFEjwsLCuHkz+wr9t2/fZvny5TRp0iRfixPCHE6fTqBlyx9Yvz4aOzst06Y1Ye3alyUMhSjijArEN998Ey8vL0aNGkVKSgpDhw7F1dWVrl275nd9QphchQpOuLra0LhxKbZufY0+fWpgYaExd1lCiHxm9JBp79696d27t2Go9P6xRCGKgk2bLtK4sQ8uLjbY2moJD2+Pp6edBKEQxYhRgRgbG5vjcWpqquHn+3ewEKIwio9PZeLEPfzyy0W6dq3MggUtAChZ0t68hQkhTM6oQPznZdb+LTw8PM+KEcKUfv75AhMn7iEhIQ17ey116nihlJLRDyGKKaMC8d+hl5iYyNq1a6lWrVq+FCVEfoqLy+4Vbtx4EYCmTX2YOzeQMmVk1rQQxdlTnVXs4uJC7969+fbbb/O6HiHyVUJCGi1b/sDGjRdxcLBi1qxmhIe3kzAUQjz9tUyvX79Oenp6XtYiRL5zc7PlxRfL8fffScyZI71CIcT/GBWI77//fo7jKunp6Vy5coUuXbrkW2FC5AWlFOvXR+Pr60j9+tkTwD78sAk2NpZyrFAIkYNRgdiqVascj21tbSlXrhylSpXKl6KEyAuxsSlMmLCbzZsvU6GCE1u2vIadnRZbW5Pd5EUIUYg89ptBr9dz4sQJBg4ciJWVlSlqEuKZKKX473/P8/77+0hMTKdECStCQmpha2tp7tKEEAXYYwPRwsKCP//8U4aXRKFw48Y9xo/fzZYtfwPQooUvs2c3p3TpEmauTAhR0Bk1y7R9+/Z8//336HS6/K5HiKeWlaWnS5df2LLlbxwdrZg7N5Cvv35RwlAIYZRce4i7d++mWbNm/PbbbyQmJrJx40acnJxyrPPZZ5/la4FCGMvS0oIxYwL44YdzfPRRM3x8JAiFEMbLNRCXLl1Ks2bNePfdd01VjxBGU0qxdu057txJp3//5wDo2LEiHTtWlCF+IcQTyzUQlVIAVK9e3STFCGGsmJh7jB27i23brmBlZUHbtuUoV85JglAI8dRyDcT7M0xzU7NmzTwtSIjcKKX4/vuzhIbu5+7dDJydrZk6tQlly8oJ9kKIZ5NrIGZmZrJkyRJDT/HfNBoNYWFh+VKYEP92/XoyY8fuYvv2qwC0aVOWWbOa4e3tYObKhBBFQa6BaGtrK4EnCoxJk/ayfftVXFxs+PDDJnTq5CdDpEKIPCOX7BCFRmhoI2xttUyZ0kjuVyiEyHO5nof4qKFSIfKbUopvvz1Dv36/o9dnfw7LlnXi009bSRgKIfJFrj3E1atXm6oOIQyuXUtm9OhIIiOvAbB9+xVaty5r5qqEEEWdDJmKAkMpxTffnOHDDw+QnJyJi4sN06c3oVWrMuYuTQhRDEggigLhypUkxozZxa5d2b3Cdu3KM2NGUzw9ZXhUCGEaEoiiQFi37jy7dl3D1dWG6dObytVmhBAmJ4EozCYzU4+VVfa8rnfeqUVSUgYDBz6Ph4edmSsTQhRHRt3tQoi8pNcrvvzyFIGB3xMfnwqAlZUF773XUMJQCGE2EojCpC5fvku3bht57709/P13EuvXR5u7JCGEAGTIVJiIXq9YteoU06cfJDVVh4eHHTNmNKV9+wrmLk0IIQAJRGECly7dZdSonezffwOAV1/148MPm+DmZmvmyoQQ4n8kEEW+i429x4EDN/D0tGPmzKa89JL0CoUQBY8EosgXCQlphh5gw4alWLSoJS1a+EqvUAhRYMmkGpGn9HrF0qXHadDgO8Ol1wA6d64kYSiEKNCkhyjyTHR0IqNGRRIVFQvArl1XCQwsbeaqhBDCOBKI4pllZelZtuwEs2cfIi0tCy8vOz76qDlt25Yzd2lCCGE0CUTxTK5dSyYkZBuHDmX3Crt0qUxoaCNcXWV4VAhRuJgsEI8dO8bKlSvR6/W0bt2aV199Ncfzv/zyC1u3bsXS0hInJyfeeecdPD09TVWeeEr29lr+/vsuJUva89FHzWjTRnqFQojCySSBqNfrWb58OZMmTcLd3Z0JEyYQEBCAr6+vYZ3y5csza9YsbGxs+P333/n6668ZMWKEKcoTT+j8+UTKlHHExsYSV1dbvvzyBcqVc8LFxcbcpQkhxFMzySzT8+fP4+3tTcmSJdFqtTRp0oSoqKgc69SsWRMbm+wv1MqVK5OQkGCK0sQT0On0zJmzn7Zt/8v8+UcMy2vV8pQwFEIUeibpISYkJODu7m547O7uzrlz5x65/rZt26hdu7YJKhPGOnv2NiNH7uTo0VtA9nmGSim5RZMQosgocJNqIiMjuXDhAqGhoQ99PiIigoiICABmzZqFh4dHnu07L7dVVOh0eubPP8DUqbvIyMjC19eJTz99kRde8DN3aQWaVquVz5ORpK2MI+2U/0wSiG5ubsTHxxsex8fH4+bm9sB6f/75J+vWrSM0NBQrK6uHbis4OJjg4GDD47i4uGeuzycPt1WUJCam06PHJo4dy+4VvvlmFRYsaEdmZrK01WN4eHhIGxlJ2so40k7G8/HxefxKD2GSY4h+fn7ExMRw8+ZNdDode/fuJSAgIMc6Fy9eZOnSpYwdOxZnZ2dTlCUew9nZGjc3W3x8HPjmmxf5+ONAnJ3ldAohRNFkkh6ipaUlffv2Zfr06ej1elq2bEmZMmUIDw/Hz8+PgIAAvv76a9LS0pg3bx6Q/d/QuHHjTFGe+IczZxKwtrakYkVnNBoN8+cHYW1tiZOTtblLE0KIfKVRSilzF/Esrl+//szb8Pku+/Ji19+49pg1i67MTD2ffvoH8+cf4fnnPVi3rgOWlg8OIMiwjXGknYwnbWUcaSfjPe2QaYGbVCNM79SpeEaOjOT48ew/turV3cnI0GNnJ9d+F0IUHxKIxVhmpp6wsGMsXHiUzEw9vr4lmDMnkObN5YLcQojiRwKxmNLrFV26/GK4BmmvXtWZOLE+JUrIsUIhRPEkgVhMWVhoaNeuPDdvpjBnTiBNmz7dmLsQQhQVEojFyIkT8Vy/nmy4LdN//lOTnj2r4eDw8HM+hRCiOJFALAYyMrJYtOgYn3xyFHt7K7Zt60KpUg5YWlrg4CATZ4QQAiQQi7wTJ+IYPnwnp09nXyz9tdcqyTmFQgjxEBKIRVR6ehYLFx4lLOwYWVmKcuUcmTs3iMaNS5m7NCGEKJAkEIuo4cN38NNPFwDo168G48fXx95ejhUKIcSjSCAWUYMGPc+pUwnMnt2Mhg2lVyiEEI8jMyqKiD/+uMWcOYcNj2vV8mTbttckDIUQwkjSQyzk0tOzmDfvCJ999gdZWYp69bxo2bIMwEOvRSqEEOLhJBALsaNHbzJy5E7Onk1Eo4GBA5+jUSPpEQohxNOQQCyE0tJ0/98r/BO9XuHn58zcuUHUr1/S3KUJIUShJYFYCIWF/cHixX9gYaHhnXeeZ9SoetjZyVsphBDPQr5FC6GBA5/7/+HSutSrJ71CIYTICzLrohA4dCiWt976jZSUTAAcHa355puXJAyFECIPSSAWYKmpOqZO3c+rr/7Etm1XWLr0hLlLEkKIIkuGTAuoqKgbjBwZyYULd7Cw0DB48PMMHPicucsSQogiSwKxgElN1fHRR1EsW3YCpaBKFVfmzQuidm1Pc5cmhBBFmgRiAbNr1zWWLj2BpaWGkJBajBhRFxsbS3OXJYQQRZ4EYgGg1yssLDQAtG1bjiFDatGuXQVq1ZJeoRBCmIpMqjGz/ftjaNXqB06ejDcsmzChgYShEEKYmASimaSkZDJ58l5ee+0Xzp1LZMmSP81dkhBCFGsyZGoGe/deZ/ToSC5fTkKr1fDuu3UYOrS2ucsSQohiTQLRhO7dy2TGjIN8+eUpAKpXd2P+/CBq1vQwc2VCCCEkEE0oISGNtWvPodVqGDasDkOG1MbaWmaQCiFEQSCBmM/u3cvEzk6LhYWGMmUcmT8/iHLlnKhZ093cpQkhhPgHmVSTj3bvvkbr1j+wevVpw7L27StIGAohRAEkgZgPkpMzGD9+N927/8qVK8n8/HM0SilzlyWEECIXMmSaxyIjrzF6dCTXriVjZWXBiBF1CQmphUajMXdpQgghciGBmEdSU3VMmbKPb745A8Dzz3swf34QVau6mbkyIYQQxpBAzCNWVhYcPx6HtbUFI0fW4513nkerlRFpIYQoLCQQn8HduxnodHrc3GzRai1YtKgFer2iShXpFQohRGEjgfiUtm+/wpgxu6hd25OlS4PRaDRUruxq7rKEMAulFGlpaej1+ic+Xh4bG0t6eno+VVZ0SDvlpJTCwsICW1vbPJujIYH4hO7cSWfq1P2sWXMWgJIl7UlOzsTR0drMlQlhPmlpaVhZWaHVPvlXilarxdJSLlDxONJOD9LpdKSlpWFnZ5cn25NAfAJbt/7N2LG7uXHjHjY2loweXY8BA56TY4Wi2NPr9U8VhkI8C61Wm6e9ZvkEG0EpxZgxu/juu78AqFPHi/nzA2WIVIj/J6cVCXPJy8+edG2MoNFocHS0xsbGksmTG7JhQwcJQyEKmDJlytCmTRtatWpFr169uHPnjuG5v/76i65du9K8eXOaNm3K/Pnzc1wsY9u2bbz00ku0aNGCtm3b8sEHH5jjJeTq+PHjjBo1ytxlPNL+/ft54YUXKFu2LL/88ssj1/vzzz9p3bo1TZs2ZfLkyYb34fbt27z++us0bdqU119/ncTERAC2bNnCxx9/bIqXIIH4KImJ6Zw48b+b9o4dG8Dvv3dm0KDnsbSUZhOioLG1tWXLli1s27YNFxcXvvzySwBSU1Pp06cPQ4YMYdeuXURERHD48GFWrVoFwJkzZ5g0aRKffPIJO3bsYNOmTZQvXz5Pa9PpdM+8jYULF9K3b1+T7vNJlC5dmvnz5/Pqq6/mut6ECROYPXs2u3fv5uLFi2zfvh2AxYsX06xZM/bs2UOzZs1YvHgxAMHBwWzZsoXU1NT8fgkSiA/z+++XadXqB/r02UxSUgYAdnZaKlVyMW9hQgij1KtXjxs3bgCwfv16AgICCAoKAsDOzo5p06YRFhYGwKeffsrQoUOpVKkSAJaWlvTq1euBbd67d48RI0bQunVrgoOD2bhxIwCVK1c2rPPLL78wfPhwAIYPH864ceN4+eWXmTZtGg0bNszRa23atCm3bt0iPj6e/v37065dO9q1a0dUVNQD+05OTubUqVPUqFEDgKNHj9KhQwfatm1Lx44dOX/+PADh4eH07t2brl270r17d1JSUhg5ciTt27enbdu2bN68GYArV67QqVMnXnjhBV544YWH7vNJlSlThurVq2Nh8ehYiY2NJSkpiXr16qHRaOjSpQu//fYbAJs3b6Zr164AdO3a1bBco9HQuHFjtmzZ8sw1Po7JjiEeO3aMlStXotfrad269QP/RWRmZhIWFsaFCxdwdHRk+PDheHl5mao8AG7fTmPKlH38+GP2h6t+/ZLcvZshM0iFeAI+35XOl+1ef+OaUetlZWWxe/du3njjDSB7uPT555/PsU758uVJSUkhKSmJv/76i4EDBz52uwsWLMDR0ZGtW7cCGIb0chMTE8OGDRuwtLREr9fz22+/0b17d44cOYKvry+enp4MHjyY/v3706BBA65du8abb77Jzp07c2znjz/+oGrVqobHlSpVYt26dWi1WiIjI/noo49YunQpkD20GhERgaurKzNnzqRp06bMmzePO3fu0L59e5o3b46Hhwffffcdtra2XLhwgcGDB7Np06YH6u/UqRPJyckPLJ88eTKBgYGPff3/duPGDUqVKmV4XKpUKcM/LnFxcZQsWRIALy8v4uLiDOvVqlWLgwcP0rFjxyfe55MwSSDq9XqWL1/OpEmTcHd3Z8KECQQEBODr62tYZ9u2bTg4OPDJJ5+wZ88evvnmG0aMGGGK8gD46WQV+s/+gZs3U7G1tWT8+Pr07VtDhkeFKCTS0tJo06YNN27coHLlyk/1hZ2bXbt28emnnxoeu7i4PPZ3Xn75ZcOpEh06dGDBggV0796dDRs2GL7cd+3axdmzZw2/k5yczL1793BwcDAsu3nzJu7u/7tLzt27dxk+fDgXL15Eo9GQmZlpeC4wMBBX1+w5DpGRkWzZsoUlS5YAkJ6ezrVr1yhZsiTvvfcep06dwsLCggsXLjy0/nXr1j32NeYHjUaTY7KMh4cHsbGx+b5fkwTi+fPn8fb2NqR/kyZNiIqKyhGIhw4dMnSXGzVqxIoVK1BKmWT22qif2jIvsgmQSoMGJZk7N4iKFZ3zfb9CFEXG9uTu02q1eXK86/4xxNTUVN58802+/PJL+vXrh7+/P/v378+x7uXLl7G3t8fR0RF/f3+OHz9uGI58Uv/8jvr3KQD29vaGnwMCArh06RLx8fFs3ryZYcOGAdkdhp9//hlbW9tcX9s/t/3xxx/TpEkTli9fzpUrV+jSpctD96mU4osvvjAMB983d+5cPD092bJlC3q9nooVKz50v3ndQ/T29iYmJsbwOCYmBm9vb+B/oVeyZEliY2Nz/AOQlpaWa/vkFZMEYkJCQo4X5+7uzrlz5x65jqWlJfb29iQlJeHk5JRjvYiICCIiIgCYNWsWHh4ez1xfq0oXWbIvgA9ntSMkJAALC5lC/iharTZP2ryoK27tFBsb+0znIebVOYxarRZHR0dmzJhB79696devH127djWMPAUFBZGamsr777/PkCFD0Gq1DBkyhL59+9K4cWP8/PzQ6/V89dVXDxxHDAoKYvXq1UybNg3IHjJ1cXHB09OTCxcuUKlSJTZv3kyJEiXQarVYWFhgaWmZ47W1a9eOqVOn4u/vbzgk1KJFC1atWsXgwYMBOHHiBDVr1syx76pVq/LFF18YtpWcnEzp0qXRarX88MMPaDQaw4n7FhYWhvVatmzJqlWrmDFjBhqNhuPHj/Pcc8+RnJyMj48P1tbWfPfdd2RlZT30Pfj555+f+D142Ou+r3Tp0jg6OnLs2DHq1avHjz/+SL9+/dBqtbz44ov8+OOPDB06lB9//JGXXnrJsI3Lly9TrVq1h27TxsYmz/7WCt15iMHBwQQHBxse/3Oc+WnV+XAHf31gh4VFKgkJ8Y9dvzjz8PDIkzYv6opbO6Wnpz/1VVTyqocI/5tZWa1aNapWrcoPP/xAly5dWLFiBZMnT2b8+PHo9Xpee+01evXqhU6no0qVKoSGhjJw4EBSU1PRaDQEBwc/UNPQoUOZOHEigYGBWFhYMHLkSNq1a8eECRPo2bMnbm5u1KpVi3v37qHT6dDr9WRlZeXYzssvv0y7du2YP3++YfnUqVOZOHEiLVq0QKfT0bBhQz766KMc+65QoQJ3794lMTGREiVKMGjQIIYPH868efNo3bo1Sil0Oh1ZWVno9XrDtocOHcqUKVNo0aIFer2eMmXKsHr1at566y0GDBhAeHg4LVu2xN7e/pnfg2PHjtGvXz/u3LnD5s2bmT17tmEGaZs2bQyTYmbMmMGIESNIS0ujZcuWBAUFodPpeOeddxg0aBDffPMNvr6+LFmyxFDTrl27mDBhwkNrTE9Pf+BvzcfH56leg0aZ4M61Z8+eZe3atbz33nvA/8alO3XqZFhn+vTpdO3aFX9/f7KyshgwYADLli177JDp9evX86TG4vYF9rSknYxT3NopJSUlx1Ddk8jLQCzKli1bhr29PW+++aa5SzGpW7duMXjwYL7//vuHPv+wz97TBqJJZoz4+fkRExPDzZs30el07N27l4CAgBzr1KtXjx07dgDZJ3jWqFFDrn4hhBD/r3fv3lhbF78Z79euXeP99983yb5M0kMEOHLkCKtWrUKv19OyZUs6d+5MeHg4fn5+BAQEkJGRQVhYGBcvXqREiRIMHz7cMAknN9JDNC1pJ+MUt3aSHmL+k3Z6uLzsIZosEPOLBKJpSTsZp7i1kwRi/pN2erhCN2QqhCjaCvn/1aIQy8vPngSiEOKZWVhYSO9FmJxOp8v1UnFPqtCddiGEKHhsbW1JS0sjPT39iSfD2djYyJ3gjSDtlJNSCgsLizw9YV8CUQjxzDQazVPftby4HW99WtJO+U+GTIUQQggkEIUQQghAAlEIIYQAisB5iEIIIURekB7i/xs/fry5SygUpJ2MI+1kPGkr40g7Ge9p20oCUQghhEACUQghhAAkEA3+eY9F8WjSTsaRdjKetJVxpJ2M97RtJZNqhBBCCKSHKIQQQgDF7NJtx44dY+XKlej1elq3bs2rr76a4/nMzEzCwsK4cOECjo6ODB8+HC8vL/MUa2aPa6tffvmFrVu3YmlpiZOTE++88w6enp7mKdaMHtdO9+3fv5958+Yxc+ZM/Pz8TFtkAWFMW+3du5e1a9ei0WgoV64cw4YNM32hZva4doqLi2Px4sXcu3cPvV7Pm2++Sd26dc1TrBl9+umnHDlyBGdnZ+bOnfvA80opVq5cydGjR7GxsSEkJISKFSvmvlFVTGRlZakhQ4aoGzduqMzMTDV69Gh15cqVHOv89ttv6vPPP1dKKbV79241b948c5Rqdsa01fHjx1VaWppSSqnNmzcXy7Yypp2UUiolJUW9//77auLEier8+fNmqNT8jGmr69evqzFjxqikpCSllFKJiYnmKNWsjGmnJUuWqM2bNyullLpy5YoKCQkxR6lmd/LkSRUdHa1Gjhz50OcPHz6spk+frvR6vfrrr7/UhAkTHrvNYjNkev78eby9vSlZsiRarZYmTZoQFRWVY51Dhw7RokULABo1asSJEyeK5X3ejGmrmjVrYmNjA0DlypVJSEgwR6lmZUw7AYSHh/PKK69gZWVlhioLBmPaauvWrbzwwguUKFECAGdnZ3OUalbGtJNGoyElJQXIvjmuq6urOUo1u+rVqxs+Kw9z6NAhAgMD0Wg0+Pv7c+/ePW7fvp3rNotNICYkJODu7m547O7u/sCX+D/XsbS0xN7enqSkJJPWWRAY01b/tG3bNmrXrm2CygoWY9rpwoULxMXFFcshrX8ypq2uX79OTEwMkydP5r333uPYsWMmrtL8jGmnrl27smvXLgYNGsTMmTPp27evqcssFBISEvDw8DA8ftz3GBSjQBT5IzIykgsXLtCxY0dzl1Lg6PV6Vq9ezdtvv23uUgoFvV5PTEwMU6ZMYdiwYXz++efcu3fP3GUVOHv27KFFixYsWbKECRMm8Mknn6DX681dVpFQbALRzc2N+Ph4w+P4+Hjc3NweuU5WVhYpKSk4OjqatM6CwJi2Avjzzz9Zt24dY8eOLZbDgY9rp7S0NK5cucIHH3zA4MGDOXfuHLNnzyY6Otoc5ZqVsX9/AQEBaLVavLy8KFWqFDExMaYu1ayMaadt27bRuHFjAPz9/cnMzCyWI1mP4+bmluP+kY/6HvunYhOIfn5+xMTEcPPmTXQ6HXv37iUgICDHOvXq1WPHjh1A9qzAGjVqPPHdv4sCY9rq4sWLLF26lLFjxxbLYz3w+Hayt7dn+fLlLF68mMWLF1O5cmXGjh1bLGeZGvOZatCgASdPngTg7t27xMTEULJkSXOUazbGtJOHhwcnTpwA4OrVq2RmZuLk5GSOcgu0gIAAIiMjUUpx9uxZ7O3tH3u8tVidmH/kyBFWrVqFXq+nZcuWdO7cmfDwcPz8/AgICCAjI4OwsDAuXrxIiRIlGD58eLH7g7zvcW314Ycf8vfff+Pi4gJk/5GOGzfOvEWbwePa6Z9CQ0N56623imUgwuPbSinF6tWrOXbsGBYWFnTu3JmmTZuau2yTe1w7Xb16lc8//5y0tDQAevbsSa1atcxctektWLCAU6dOkZSUhLOzM926dUOn0wHQtm1blFIsX76cP/74A2tra0JCQh77t1esAlEIIYR4lGIzZCqEEELkRgJRCCGEQAJRCCGEACQQhRBCCEACUQghhAAkEIXIVWhoKFu3bjV3GbnatWsX06ZNe+Tzp0+fLpZ3jRDiSclpF6LYGDx4MImJiVhY/O//wIULF+Z69YrQ0FCaN29O69at86yO0NBQzp07h4WFBdbW1lSrVo1+/frl2UWau3XrxqJFi/D29s6T7T3K999/z7p169BqtVhaWuLr68vbb7+Nv79/gapTCGMVq/shCjFu3Dief/55c5dB3759ad26NcnJycydO5dVq1YxfPhwc5f1xBo3bszQoUPJysri+++/Z968eSxZssTcZQnxVCQQRbGWnJxMWFgY586dQ6/XU6VKFfr375/jjgP33bhxg88++4xLly6h1WqpWbMmI0aMAODatWusWLGCCxcu4OTkRPfu3WnSpMlj91+iRAkaNmzIli1bAPjrr7/48ssvuX79Oj4+PvTu3ZsqVaoAsGPHDn744Qfu3r2Lo6Mjr7/+Os2bN2fHjh1s3bqVDz/8kClTpgAwZswYAN555x2cnZ355JNPWLJkCevXryc6OppRo0YZali5ciVKKfr27UtKSgqrVq3i6NGjaDQaWrZsSbdu3XL0qh/G0tKS5s2bs27dOu7evYuTkxPnz59n5cqVXLt2DWtraxo2bEivXr3QarUPrbNJkyYcPnyYNWvWcOvWLXx9fenfvz/lypV7bDsKkRckEEWxppSiRYsWjBgxAr1ez2effcby5csZO3bsA+uuWbOGWrVqMWXKFHQ6HRcuXACyL+I9bdo0unXrxsSJE/n777+ZNm0aZcuWxdfXN9f93717lwMHDlC+fHmSk5OZNWsWffr0oWnTpuzbt49Zs2axaNEirKysWLlyJTNnzsTHx4fbt2+TnJz8wPY++OADunXrxscff2wYirx/fVCApk2b8sMPP5CamoqdnR16vZ59+/YxevRoABYvXoyzszOLFi0iPT2dWbNm4e7uTps2bXJ9HTqdjp07d+Lo6IiDgwMAFhYW9OrVCz8/P+Lj45k5cyabN2+mffv2D63z4sWLfPbZZ4wbNw4/Pz8iIyOZPXs2CxYsKJYXjxemJ5NqRLHy8ccf07t3b3r37s3s2bNxdHSkUaNG2NjYYGdnR+fOnTl9+vRDf1er1XLr1i1u376NtbU1VatWBbKvPenp6UnLli2xtLSkQoUKNGzYkH379j2yjpUrV9K7d2/GjBmDq6srvXr14siRI3h7exMYGIilpSXNmjXDx8eHw4cPA9k3hv3777/JyMjA1dWVMmXKPPHr9/T0pEKFChw8eBCAEydOYGNjg7+/P4mJiRw9epTevXtja2uLs7Mz7du3Z+/evY/c3r59++jduzc9evRg69atjBw5EktLSwAqVqyIv78/lpaWeHl5ERwczKlTpx65rYiICIKDg6lcuTIWFha0aNECrVbLuXPnnvh1CvE0pIcoipUxY8bkOIaYnp7OqlWrOHbsmOHee6mpqej1+geGCXv27MmaNWuYOHEiDg4OvPzyy7Rq1Ypbt25x7tw5evfubVg3KyuLwMDAR9bRp0+fBybqJCQk4OnpmWOZp6cnCQkJ2NraMnz4cH7++WeWLFlClSpVePvttylduvQTt0GzZs3Ys2cPQUFB7N6923AB7bi4OLKyshgwYIBhXaXUQ4eP77t/DPHu3bvMnTuXCxcuUKNGDSD7hr+rV68mOjqajIwMsrKyqFix4iO3FRcXx86dO/ntt98My3Q63WNv6ipEXpFAFMXazz//zPXr15kxYwYuLi5cunSJsWPH8rDJ1y4uLgwaNAiAM2fO8OGHH1K9enXc3d2pXr06kydPfqZa3NzcOHDgQI5lcXFx1K5dG4DatWtTu3ZtMjIyWLNmDZ9//jlTp0594v00btyY1atXEx8fz8GDBw2nbLi7u6PValm+fLmhl2csJycnBg4cyPjx42nWrBmurq4sW7aM8uXLM2zYMOzs7Ni4cSP79+9/5Dbc3d3p3LkznTt3fuLXJERekCFTUaylpaVhbW2Nvb09ycnJrF279pHr7tu3z3Dz1vvHyTQaDfXq1SMmJobIyEh0Oh06nY7z589z9erVJ6qlTp06xMTEsHv3brKysti7dy9Xr16lbt26JCYmEhUVRVpaGlqtFltb20feq9PZ2ZnY2NhH7sfJyYkaNWrw6aef4uXlZTjO6erqSq1atVi9ejUpKSno9Xpu3LiR6zDnP/n4+FCrVi02bNgAZPe07e3tsbW15dq1a/z++++51tm6dWu2bNnCuXPnUEqRlpbGkSNHSE1NNWr/Qjwr6SGKYq1du3YsWrSIfv364ebmxssvv0xUVNRD142OjubLL78kJSUFFxcX+vTpY7hf5qRJk1i1ahWrVq1CKUW5cuXo1avXE9Xi6OjI+PHjWblyJUuXLsXb25vx48fj5OTE7du3+eWXXwgLC0Oj0VC+fHn69+//0O107dqVxYsXk5GRwYABAx56A+dmzZoRFhZGz549cywfMmQI33zzDSNHjiQ1NZWSJUvyyiuvGP0aOnbsyNSpU+nUqRNvvfUWX3zxBRs2bKBChQo0adLEcGPbh9XZpEkTBg4cyIoVK4iJiTEcp61WrZrR+xfiWciJ+UIIIQQyZCqEEEIAEohCCCEEIIEohBBCABKIQgghBCCBKIQQQgASiEIIIQQggSiEEEIAEohCCCEEIIEohBBCAPB/lXpI3lOBtdsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 504x360 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 绘制ROC曲线：\n",
    "from sklearn.metrics import roc_curve,auc\n",
    "from sklearn.metrics import make_scorer,f1_score\n",
    "scorer = make_scorer(f1_score,pos_label=0)\n",
    "gs = GridSearchCV(estimator=pipe_svc,param_grid=param_grid,scoring=scorer,cv=10)\n",
    "y_pred = gs.fit(X_train,y_train).decision_function(X_test)\n",
    "#y_pred = gs.predict(X_test)\n",
    "fpr,tpr,threshold = roc_curve(y_test, y_pred) ###计算真阳率和假阳率\n",
    "roc_auc = auc(fpr,tpr) ###计算auc的值\n",
    "plt.figure()\n",
    "lw = 2\n",
    "plt.figure(figsize=(7,5))\n",
    "plt.plot(fpr, tpr, color='darkorange',\n",
    "         lw=lw, label='ROC curve (area = %0.2f)' % roc_auc) ###假阳率为横坐标，真阳率为纵坐标做曲线\n",
    "plt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')\n",
    "plt.xlim([-0.05, 1.0])\n",
    "plt.ylim([-0.05, 1.05])\n",
    "plt.xlabel('False Positive Rate')\n",
    "plt.ylabel('True Positive Rate')\n",
    "plt.title('Receiver operating characteristic ')\n",
    "plt.legend(loc=\"lower right\")\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3.结语"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "本章中，我们重点讨论了各种回归和分类算法的具体推导与简单应用，并且给出了如何使用sklearn这个强大的python工具库进行简单的机器学习模型的建模代码。本章的重点是各个基础算法的掌握，包括回归和分类(重点是分类)算法以及怎么用网格搜索以及其他搜索方式进行调参。简单模型在进行复杂项目的时候往往显得力不从心，那么在下一章中，我们将开始本次开源项目的主题----集成学习，我们着重讨论如何将本章所学的基础模型进行集成，变成功能更加强大的集成模型。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了巩固本章的理解，在这里给个小任务，大家结合sklearn的fetch_lfw_people数据集，进行一次实战。fetch_lfw_people数据集是一个图像数据集，详细内容可以参照：                   \n",
    "https://scikit-learn.org/stable/modules/generated/sklearn.datasets.fetch_lfw_people.html                   \n",
    "案例的内容是对图像进行识别并分类。               \n",
    "参考资料：                \n",
    "https://blog.csdn.net/cwlseu/article/details/52356665                      \n",
    "https://blog.csdn.net/jasonzhoujx/article/details/81905923"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.0"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
